Fork me on GitHub

idea中Spark on yarn 远程Debug Spark 源码(Yarn Client+attach 模式)

发表于 | 分类于 |
| 字数统计: 24,968 | 阅读时长 ≈ 154

背景

idea 中连接yarn集群debug spark代码,有利于定位线上问题。

环境

默认你hadoop环境已经搞定情况下的说明！！！
scala版本： 2.11.8
jdk版本：JDK1.8
hadoop版本： 2.7.3
spark版本： 2.4.1
zookeeper： 3.4.6
高能预警： 一定要保持yarn集群机器scala版本和项目pom文件中scala版本一致。

linux版本 IP hostname 进程
centos7 192.168.8.81 hadoop01 NameNode、DFSZKFailoverController
centos7 192.168.8.82 hadoop02 NameNode、DFSZKFailoverController、ResourceManager、JournalNode、NodeManager、DataNode、QuorumPeerMain
centos7 192.168.8.83 hadoop03 JournalNode、NodeManager、DataNode、QuorumPeerMain
centos7 192.168.8.84 hadoop04 JournalNode、NodeManager、DataNode、QuorumPeerMain

idea所在机器：

linux版本 IP hostname
ubuntu18.04 192.168.8.85 hadoop05

maven项目中resources目录中配置文件

高能预警： 配置文件需要和hadoop集群一致

具体配置文件如下：
core-site.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <!-- 指定hdfs的nameservice为ns1 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://ns1/</value>
    </property>
    <!-- 指定hadoop临时目录 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/hadoop/hadoop/tmp</value>
    </property>
    <!-- 指定zookeeper地址 -->
    <property>
        <name>ha.zookeeper.quorum</name>
        <value>hadoop02:2181,hadoop03:2181,hadoop04:2181</value>
    </property>
</configuration>

hdfs-site.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<!--指定hdfs的nameservice为ns1，需要和core-site.xml中的保持一致 -->
<property>
<name>dfs.nameservices</name>
<value>ns1</value>
</property>
<!-- ns1下面有两个NameNode，分别是nn1，nn2 -->
<property>
<name>dfs.ha.namenodes.ns1</name>
<value>nn1,nn2</value>
</property>
<!-- nn1的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.ns1.nn1</name>
<value>hadoop01:9000</value>
</property>
<!-- nn1的http通信地址 -->
<property>
<name>dfs.namenode.http-address.ns1.nn1</name>
<value>hadoop01:50070</value>
</property>
<!-- nn2的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.ns1.nn2</name>
<value>hadoop02:9000</value>
</property>
<!-- nn2的http通信地址 -->
<property>
<name>dfs.namenode.http-address.ns1.nn2</name>
<value>hadoop02:50070</value>
</property>
<!-- 指定NameNode的元数据在JournalNode上的存放位置 -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://hadoop02:8485;hadoop03:8485;hadoop04:8485/ns1</value>
</property>
<!-- 指定JournalNode在本地磁盘存放数据的位置 -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/home/hadoop/journaldata</value>
</property>
<!-- 开启NameNode失败自动切换 -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<!-- 配置失败自动切换实现方式 -->
<property>
<name>dfs.client.failover.proxy.provider.ns1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!-- 配置隔离机制方法，多个机制用换行分割，即每个机制暂用一行-->
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property>
<!-- 使用sshfence隔离机制时需要ssh免登陆 -->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/hadoop/.ssh/id_rsa</value>
</property>
<!-- 配置sshfence隔离机制超时时间 -->
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>
<!--修改block块的大小,默认为128M-->
<property>
<name>dfs.block.seze</name>
<value>128</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
</configuration>

yarn-site.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>
    <!--开启RM高可用 -->
    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
        <value>true</value>
    </property>
    <!-- 指定RM的cluster id -->
    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>yrc</value>
    </property>
    <!-- 指定RM的名字 -->
    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
    </property>
    <!-- 分别指定RM的地址 -->
    <property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>hadoop02</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>hadoop03</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address.rm1</name>
        <value>hadoop02:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address.rm2</name>
        <value>hadoop03:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address.rm1</name>
        <value>hadoop02:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address.rm2</name>
        <value>hadoop03:8030</value>
    </property>
    <!-- 指定zk集群地址 -->
    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>hadoop02:2181,hadoop03:2181,hadoop04:2181</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
        <value>hadoop02:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
        <value>hadoop03:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address.rm1</name>
        <value>hadoop02:8088</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address.rm2</name>
        <value>hadoop03:8088</value>
    </property>
    <property>
        <name>yarn.client.failover-proxy-provider</name>
        <value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value>
    </property>
    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
    </property>
    <property>
        <!-- 两个可选值：org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore 以及 默认值org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore -->
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>hadoop02:8088</value>
    </property>
    <property>
        <name>yarn.web-proxy.address</name>
        <value>192.168.8.82:8089</value>
    </property>
    <property>
        <name>yarn.log-aggregation-enable</name>
        <!-- 打开日志聚合功能，这样才能从web界面查看日志 -->
        <value>true</value>
    </property>
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <!-- 聚合日志最长保留时间 -->
        <value>86400</value>
    </property>
    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <!-- 中间结果存放位置。注意，这个参数通常会配置多个目录，已分摊磁盘IO负载。 -->
        <value>/home/hadoop/hadoop/data/localdir1,/home/hadoop/hadoop/data/localdir2</value>
    </property>
    <property>
        <name>yarn.nodemanager.log-dirs</name>
        <!-- 日志存放位置。注意，这个参数通常会配置多个目录，已分摊磁盘IO负载。 -->
        <value>/home/hadoop/hadoop/data/hdfs/logdir1,/home/hadoop/hadoop/data/hdfs/logdir2</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    </property>
</configuration>

spark-defaults.conf配置文件：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# Default system properties included when running spark-submit.
# This is useful for setting default environmental settings.
spark.driver.extraClassPath         /home/hadoop/spark/jars/*:/home/hadoop/hadoop/share/hadoop/hdfs/*:/home/hadoop/hadoop/share/hadoop/common/*:/home/hadoop/hadoop/share/hadoop/common/lib/*:/home/hadoop/hadoop/share/hadoop/yarn/*:/home/hadoop/hadoop/share/hadoop/yarn/lib/*

spark.executor.extraClassPath     /home/hadoop/spark/jars/*:/home/hadoop/hadoop/share/hadoop/hdfs/*:/home/hadoop/hadoop/share/hadoop/common/*:/home/hadoop/hadoop/share/hadoop/common/lib/*:/home/hadoop/hadoop/share/hadoop/yarn/*:/home/hadoop/hadoop/share/hadoop/yarn/lib/*
spark.driver.extraJavaOptions     -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005
spark.executor.extraJavaOptions   -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005
spark.driver.memory                512m
spark.executor.memory              512m
# Example:
# spark.master                     spark://master:7077
# spark.eventLog.enabled           true
# spark.eventLog.dir               hdfs://namenode:8021/directory
# spark.serializer                 org.apache.spark.serializer.KryoSerializer
# spark.driver.memory              5g
# spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

maven pom配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.xh.spark</groupId>
    <artifactId>sparklearning</artifactId>
    <version>1.0-SNAPSHOT</version>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <encoding>UTF-8</encoding>
        <scala.version>2.11.8</scala.version>
        <scala.binary.version>2.11</scala.binary.version>
        <spark.version>2.4.1</spark.version>
        <hadoop.version>2.7.3</hadoop.version>
        <mysql.version>5.1.35</mysql.version>
        <hive.version>2.3.3</hive.version>
        <guava.version>26.0-jre</guava.version>
        <fastjson.version>1.2.40</fastjson.version>
        <zookeeper.version>3.4.6</zookeeper.version>
    </properties>

    <dependencies>
        <!--scala-->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
            <version>${zookeeper.version}</version>
        </dependency>
        <!--spark-->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-yarn_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka-0-10_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>${fastjson.version}</version>
        </dependency>
        <!--hive-->
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-jdbc</artifactId>
            <version>${hive.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-exec</artifactId>
            <version>${hive.version}</version>
        </dependency>
        <!--hadoop-->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <!--mysql-->
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>${mysql.version}</version>
        </dependency>
        <!--guava-->
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>${guava.version}</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <!-- This plugin compiles Scala files -->
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.2</version>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <!-- This plugin compiles Java files -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <!--跳过test begin-->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.20</version>
                <configuration>
                    <skipTests>true</skipTests>
                </configuration>
            </plugin>
            <!-- This plugin adds all dependencies to JAR file during 'package' command.
            Pay EXTRA attention to the 'mainClass' tag.
            You have to set name of class with entry point to program ('main' method) -->
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <archive>
                        <manifestFile></manifestFile>
                    </archive>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <transformers>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>com.defonds.RsaEncryptor</mainClass>
                                </transformer>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>META-INF/spring.handlers</resource>
                                </transformer>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>META-INF/spring.schemas</resource>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

开发程序

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
package com.xh.spark.sql.function

import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession

object WindowFunctionTest {
  def main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf()
      .setAppName("WindowFunctionTest")
          .set("spark.master", "yarn")
          .set("spark.submit.deployMode", "client") // 部署模式为client
          .set("yarn.resourcemanager.hostname", "hadoop02") // resourcemanager主机名
          .set("spark.executor.instances", "2") // Executor实例的数量
          .set("spark.dynamicAllocation.enabled", "false")
          .setJars(List("/home/hadoop/worker/sparklearning/target/sparklearning-1.0-SNAPSHOT.jar",
            "/home/hadoop/worker/sparklearning/target/sparklearning-1.0-SNAPSHOT-jar-with-dependencies.jar"
          ))

    val spark = SparkSession
      .builder()
      .config(sparkConf)
      .getOrCreate()

    import org.apache.spark.sql.expressions.Window
    import org.apache.spark.sql.functions._
    import spark.implicits._
    val df = List(
      ("站点1", "2018-01-01", 50),
      ("站点1", "2018-01-02", 45),
      ("站点1", "2018-01-03", 55),
      ("站点2", "2018-01-01", 25),
      ("站点2", "2018-01-02", 29),
      ("站点2", "2018-01-03", 27)
    ).toDF("site", "date", "user_cnt")

    // 简单移动平均值
    // API方式
    // 窗口定义从 -1(前一行)到 1(后一行)	，每一个滑动的窗口总用有3行
    val movinAvgSpec = Window.partitionBy("site").orderBy("date").rowsBetween(-1, 1)
    df.withColumn("MovingAvg", avg(df("user_cnt")).over(movinAvgSpec)).show()

    // sql方式
     df.createOrReplaceTempView("site_info")
        spark.sql(
          """
            |select site,
            |       date,
            |       user_cnt,
            |       avg(user_cnt) over(partition by site order by date rows between 1 preceding and 1 following) as moving_avg
            |from   site_info
          """.stripMargin).show()
    
    spark.stop()
  }
}

以上确认无误之后，直接在idea里面run该程序。

执行结果如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
/usr/lib/jvm/java-1.8.0-openjdk/bin/java -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:45315,suspend=y,server=n -javaagent:/home/hadoop/idea/lib/rt/debugger-agent.jar -Dfile.encoding=UTF-8 -classpath /usr/lib/jvm/java-1.8.0-openjdk/jre/lib/charsets.jar:/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/ext/cldrdata.jar:/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/ext/jaccess.jar:/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/ext/nashorn.jar:/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/jce.jar:/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/jsse.jar:/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/management-agent.jar:/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/resources.jar:/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/rt.jar:/home/hadoop/worker/sparklearning/target/classes:/home/hadoop/repository/org/scala-lang/scala-library/2.11.8/scala-library-2.11.8.jar:/home/hadoop/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar:/home/hadoop/repository/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar:/home/hadoop/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar:/home/hadoop/repository/log4j/log4j/1.2.16/log4j-1.2.16.jar:/home/hadoop/repository/jline/jline/0.9.94/jline-0.9.94.jar:/home/hadoop/repository/junit/junit/3.8.1/junit-3.8.1.jar:/home/hadoop/repository/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.jar:/home/hadoop/repository/org/apache/spark/spark-yarn_2.11/2.4.1/spark-yarn_2.11-2.4.1.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-yarn-api/2.6.5/hadoop-yarn-api-2.6.5.jar:/home/hadoop/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/home/hadoop/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-yarn-common/2.6.5/hadoop-yarn-common-2.6.5.jar:/home/hadoop/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/home/hadoop/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/home/hadoop/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/home/hadoop/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar:/home/hadoop/repository/org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar:/home/hadoop/repository/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar:/home/hadoop/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/home/hadoop/repository/com/google/inject/extensions/guice-servlet/3.0/guice-servlet-3.0.jar:/home/hadoop/repository/com/google/inject/guice/3.0/guice-3.0.jar:/home/hadoop/repository/javax/inject/javax.inject/1/javax.inject-1.jar:/home/hadoop/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.6.5/hadoop-yarn-server-web-proxy-2.6.5.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-yarn-server-common/2.6.5/hadoop-yarn-server-common-2.6.5.jar:/home/hadoop/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-yarn-client/2.6.5/hadoop-yarn-client-2.6.5.jar:/home/hadoop/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/home/hadoop/repository/org/apache/spark/spark-core_2.11/2.4.1/spark-core_2.11-2.4.1.jar:/home/hadoop/repository/com/thoughtworks/paranamer/paranamer/2.8/paranamer-2.8.jar:/home/hadoop/repository/org/apache/avro/avro/1.8.2/avro-1.8.2.jar:/home/hadoop/repository/org/tukaani/xz/1.5/xz-1.5.jar:/home/hadoop/repository/org/apache/avro/avro-mapred/1.8.2/avro-mapred-1.8.2-hadoop2.jar:/home/hadoop/repository/org/apache/avro/avro-ipc/1.8.2/avro-ipc-1.8.2.jar:/home/hadoop/repository/com/twitter/chill_2.11/0.9.3/chill_2.11-0.9.3.jar:/home/hadoop/repository/com/esotericsoftware/kryo-shaded/4.0.2/kryo-shaded-4.0.2.jar:/home/hadoop/repository/com/esotericsoftware/minlog/1.3.0/minlog-1.3.0.jar:/home/hadoop/repository/org/objenesis/objenesis/2.5.1/objenesis-2.5.1.jar:/home/hadoop/repository/com/twitter/chill-java/0.9.3/chill-java-0.9.3.jar:/home/hadoop/repository/org/apache/xbean/xbean-asm6-shaded/4.8/xbean-asm6-shaded-4.8.jar:/home/hadoop/repository/org/apache/spark/spark-launcher_2.11/2.4.1/spark-launcher_2.11-2.4.1.jar:/home/hadoop/repository/org/apache/spark/spark-kvstore_2.11/2.4.1/spark-kvstore_2.11-2.4.1.jar:/home/hadoop/repository/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar:/home/hadoop/repository/com/fasterxml/jackson/core/jackson-core/2.6.7/jackson-core-2.6.7.jar:/home/hadoop/repository/com/fasterxml/jackson/core/jackson-annotations/2.6.7/jackson-annotations-2.6.7.jar:/home/hadoop/repository/org/apache/spark/spark-network-common_2.11/2.4.1/spark-network-common_2.11-2.4.1.jar:/home/hadoop/repository/org/apache/spark/spark-network-shuffle_2.11/2.4.1/spark-network-shuffle_2.11-2.4.1.jar:/home/hadoop/repository/org/apache/spark/spark-unsafe_2.11/2.4.1/spark-unsafe_2.11-2.4.1.jar:/home/hadoop/repository/javax/activation/activation/1.1.1/activation-1.1.1.jar:/home/hadoop/repository/org/apache/curator/curator-recipes/2.6.0/curator-recipes-2.6.0.jar:/home/hadoop/repository/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:/home/hadoop/repository/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar:/home/hadoop/repository/org/apache/commons/commons-math3/3.4.1/commons-math3-3.4.1.jar:/home/hadoop/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/home/hadoop/repository/org/slf4j/jul-to-slf4j/1.7.16/jul-to-slf4j-1.7.16.jar:/home/hadoop/repository/org/slf4j/jcl-over-slf4j/1.7.16/jcl-over-slf4j-1.7.16.jar:/home/hadoop/repository/com/ning/compress-lzf/1.0.3/compress-lzf-1.0.3.jar:/home/hadoop/repository/org/xerial/snappy/snappy-java/1.1.7.1/snappy-java-1.1.7.1.jar:/home/hadoop/repository/org/lz4/lz4-java/1.4.0/lz4-java-1.4.0.jar:/home/hadoop/repository/com/github/luben/zstd-jni/1.3.2-2/zstd-jni-1.3.2-2.jar:/home/hadoop/repository/org/roaringbitmap/RoaringBitmap/0.5.11/RoaringBitmap-0.5.11.jar:/home/hadoop/repository/commons-net/commons-net/3.1/commons-net-3.1.jar:/home/hadoop/repository/org/json4s/json4s-jackson_2.11/3.5.3/json4s-jackson_2.11-3.5.3.jar:/home/hadoop/repository/org/json4s/json4s-core_2.11/3.5.3/json4s-core_2.11-3.5.3.jar:/home/hadoop/repository/org/json4s/json4s-ast_2.11/3.5.3/json4s-ast_2.11-3.5.3.jar:/home/hadoop/repository/org/json4s/json4s-scalap_2.11/3.5.3/json4s-scalap_2.11-3.5.3.jar:/home/hadoop/repository/org/scala-lang/modules/scala-xml_2.11/1.0.6/scala-xml_2.11-1.0.6.jar:/home/hadoop/repository/org/glassfish/jersey/core/jersey-client/2.22.2/jersey-client-2.22.2.jar:/home/hadoop/repository/javax/ws/rs/javax.ws.rs-api/2.0.1/javax.ws.rs-api-2.0.1.jar:/home/hadoop/repository/org/glassfish/hk2/hk2-api/2.4.0-b34/hk2-api-2.4.0-b34.jar:/home/hadoop/repository/org/glassfish/hk2/hk2-utils/2.4.0-b34/hk2-utils-2.4.0-b34.jar:/home/hadoop/repository/org/glassfish/hk2/external/aopalliance-repackaged/2.4.0-b34/aopalliance-repackaged-2.4.0-b34.jar:/home/hadoop/repository/org/glassfish/hk2/external/javax.inject/2.4.0-b34/javax.inject-2.4.0-b34.jar:/home/hadoop/repository/org/glassfish/hk2/hk2-locator/2.4.0-b34/hk2-locator-2.4.0-b34.jar:/home/hadoop/repository/org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.jar:/home/hadoop/repository/org/glassfish/jersey/core/jersey-common/2.22.2/jersey-common-2.22.2.jar:/home/hadoop/repository/javax/annotation/javax.annotation-api/1.2/javax.annotation-api-1.2.jar:/home/hadoop/repository/org/glassfish/jersey/bundles/repackaged/jersey-guava/2.22.2/jersey-guava-2.22.2.jar:/home/hadoop/repository/org/glassfish/hk2/osgi-resource-locator/1.0.1/osgi-resource-locator-1.0.1.jar:/home/hadoop/repository/org/glassfish/jersey/core/jersey-server/2.22.2/jersey-server-2.22.2.jar:/home/hadoop/repository/org/glassfish/jersey/media/jersey-media-jaxb/2.22.2/jersey-media-jaxb-2.22.2.jar:/home/hadoop/repository/javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar:/home/hadoop/repository/org/glassfish/jersey/containers/jersey-container-servlet/2.22.2/jersey-container-servlet-2.22.2.jar:/home/hadoop/repository/org/glassfish/jersey/containers/jersey-container-servlet-core/2.22.2/jersey-container-servlet-core-2.22.2.jar:/home/hadoop/repository/io/netty/netty-all/4.1.17.Final/netty-all-4.1.17.Final.jar:/home/hadoop/repository/com/clearspring/analytics/stream/2.7.0/stream-2.7.0.jar:/home/hadoop/repository/io/dropwizard/metrics/metrics-core/3.1.5/metrics-core-3.1.5.jar:/home/hadoop/repository/io/dropwizard/metrics/metrics-jvm/3.1.5/metrics-jvm-3.1.5.jar:/home/hadoop/repository/io/dropwizard/metrics/metrics-json/3.1.5/metrics-json-3.1.5.jar:/home/hadoop/repository/io/dropwizard/metrics/metrics-graphite/3.1.5/metrics-graphite-3.1.5.jar:/home/hadoop/repository/com/fasterxml/jackson/core/jackson-databind/2.6.7.1/jackson-databind-2.6.7.1.jar:/home/hadoop/repository/com/fasterxml/jackson/module/jackson-module-scala_2.11/2.6.7.1/jackson-module-scala_2.11-2.6.7.1.jar:/home/hadoop/repository/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar:/home/hadoop/repository/com/fasterxml/jackson/module/jackson-module-paranamer/2.7.9/jackson-module-paranamer-2.7.9.jar:/home/hadoop/repository/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar:/home/hadoop/repository/oro/oro/2.0.8/oro-2.0.8.jar:/home/hadoop/repository/net/razorvine/pyrolite/4.13/pyrolite-4.13.jar:/home/hadoop/repository/net/sf/py4j/py4j/0.10.7/py4j-0.10.7.jar:/home/hadoop/repository/org/apache/spark/spark-tags_2.11/2.4.1/spark-tags_2.11-2.4.1.jar:/home/hadoop/repository/org/apache/commons/commons-crypto/1.0.0/commons-crypto-1.0.0.jar:/home/hadoop/repository/org/apache/spark/spark-sql_2.11/2.4.1/spark-sql_2.11-2.4.1.jar:/home/hadoop/repository/com/univocity/univocity-parsers/2.7.3/univocity-parsers-2.7.3.jar:/home/hadoop/repository/org/apache/spark/spark-sketch_2.11/2.4.1/spark-sketch_2.11-2.4.1.jar:/home/hadoop/repository/org/apache/spark/spark-catalyst_2.11/2.4.1/spark-catalyst_2.11-2.4.1.jar:/home/hadoop/repository/org/codehaus/janino/janino/3.0.9/janino-3.0.9.jar:/home/hadoop/repository/org/codehaus/janino/commons-compiler/3.0.9/commons-compiler-3.0.9.jar:/home/hadoop/repository/org/antlr/antlr4-runtime/4.7/antlr4-runtime-4.7.jar:/home/hadoop/repository/org/apache/orc/orc-core/1.5.5/orc-core-1.5.5-nohive.jar:/home/hadoop/repository/org/apache/orc/orc-shims/1.5.5/orc-shims-1.5.5.jar:/home/hadoop/repository/io/airlift/aircompressor/0.10/aircompressor-0.10.jar:/home/hadoop/repository/org/apache/orc/orc-mapreduce/1.5.5/orc-mapreduce-1.5.5-nohive.jar:/home/hadoop/repository/org/apache/parquet/parquet-column/1.10.1/parquet-column-1.10.1.jar:/home/hadoop/repository/org/apache/parquet/parquet-common/1.10.1/parquet-common-1.10.1.jar:/home/hadoop/repository/org/apache/parquet/parquet-encoding/1.10.1/parquet-encoding-1.10.1.jar:/home/hadoop/repository/org/apache/parquet/parquet-hadoop/1.10.1/parquet-hadoop-1.10.1.jar:/home/hadoop/repository/org/apache/parquet/parquet-format/2.4.0/parquet-format-2.4.0.jar:/home/hadoop/repository/org/apache/parquet/parquet-jackson/1.10.1/parquet-jackson-1.10.1.jar:/home/hadoop/repository/org/apache/arrow/arrow-vector/0.10.0/arrow-vector-0.10.0.jar:/home/hadoop/repository/org/apache/arrow/arrow-format/0.10.0/arrow-format-0.10.0.jar:/home/hadoop/repository/org/apache/arrow/arrow-memory/0.10.0/arrow-memory-0.10.0.jar:/home/hadoop/repository/com/carrotsearch/hppc/0.7.2/hppc-0.7.2.jar:/home/hadoop/repository/com/vlkan/flatbuffers/1.2.0-3f79e055/flatbuffers-1.2.0-3f79e055.jar:/home/hadoop/repository/org/apache/spark/spark-mllib_2.11/2.4.1/spark-mllib_2.11-2.4.1.jar:/home/hadoop/repository/org/scala-lang/modules/scala-parser-combinators_2.11/1.1.0/scala-parser-combinators_2.11-1.1.0.jar:/home/hadoop/repository/org/apache/spark/spark-graphx_2.11/2.4.1/spark-graphx_2.11-2.4.1.jar:/home/hadoop/repository/com/github/fommil/netlib/core/1.1.2/core-1.1.2.jar:/home/hadoop/repository/net/sourceforge/f2j/arpack_combined_all/0.1/arpack_combined_all-0.1.jar:/home/hadoop/repository/org/apache/spark/spark-mllib-local_2.11/2.4.1/spark-mllib-local_2.11-2.4.1.jar:/home/hadoop/repository/org/scalanlp/breeze_2.11/0.13.2/breeze_2.11-0.13.2.jar:/home/hadoop/repository/org/scalanlp/breeze-macros_2.11/0.13.2/breeze-macros_2.11-0.13.2.jar:/home/hadoop/repository/net/sf/opencsv/opencsv/2.3/opencsv-2.3.jar:/home/hadoop/repository/com/github/rwl/jtransforms/2.4.0/jtransforms-2.4.0.jar:/home/hadoop/repository/org/spire-math/spire_2.11/0.13.0/spire_2.11-0.13.0.jar:/home/hadoop/repository/org/spire-math/spire-macros_2.11/0.13.0/spire-macros_2.11-0.13.0.jar:/home/hadoop/repository/org/typelevel/machinist_2.11/0.6.1/machinist_2.11-0.6.1.jar:/home/hadoop/repository/com/chuusai/shapeless_2.11/2.3.2/shapeless_2.11-2.3.2.jar:/home/hadoop/repository/org/typelevel/macro-compat_2.11/1.1.1/macro-compat_2.11-1.1.1.jar:/home/hadoop/repository/org/apache/spark/spark-hive_2.11/2.4.1/spark-hive_2.11-2.4.1.jar:/home/hadoop/repository/com/twitter/parquet-hadoop-bundle/1.6.0/parquet-hadoop-bundle-1.6.0.jar:/home/hadoop/repository/org/spark-project/hive/hive-exec/1.2.1.spark2/hive-exec-1.2.1.spark2.jar:/home/hadoop/repository/javolution/javolution/5.5.1/javolution-5.5.1.jar:/home/hadoop/repository/log4j/apache-log4j-extras/1.2.17/apache-log4j-extras-1.2.17.jar:/home/hadoop/repository/com/googlecode/javaewah/JavaEWAH/0.3.2/JavaEWAH-0.3.2.jar:/home/hadoop/repository/org/iq80/snappy/snappy/0.2/snappy-0.2.jar:/home/hadoop/repository/org/spark-project/hive/hive-metastore/1.2.1.spark2/hive-metastore-1.2.1.spark2.jar:/home/hadoop/repository/com/jolbox/bonecp/0.8.0.RELEASE/bonecp-0.8.0.RELEASE.jar:/home/hadoop/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/home/hadoop/repository/org/datanucleus/datanucleus-api-jdo/3.2.6/datanucleus-api-jdo-3.2.6.jar:/home/hadoop/repository/org/datanucleus/datanucleus-rdbms/3.2.9/datanucleus-rdbms-3.2.9.jar:/home/hadoop/repository/commons-pool/commons-pool/1.5.4/commons-pool-1.5.4.jar:/home/hadoop/repository/commons-dbcp/commons-dbcp/1.4/commons-dbcp-1.4.jar:/home/hadoop/repository/javax/jdo/jdo-api/3.0.1/jdo-api-3.0.1.jar:/home/hadoop/repository/javax/transaction/jta/1.1/jta-1.1.jar:/home/hadoop/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/home/hadoop/repository/org/apache/calcite/calcite-avatica/1.2.0-incubating/calcite-avatica-1.2.0-incubating.jar:/home/hadoop/repository/org/apache/calcite/calcite-core/1.2.0-incubating/calcite-core-1.2.0-incubating.jar:/home/hadoop/repository/org/apache/calcite/calcite-linq4j/1.2.0-incubating/calcite-linq4j-1.2.0-incubating.jar:/home/hadoop/repository/net/hydromatic/eigenbase-properties/1.1.5/eigenbase-properties-1.1.5.jar:/home/hadoop/repository/org/apache/httpcomponents/httpclient/4.5.6/httpclient-4.5.6.jar:/home/hadoop/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/home/hadoop/repository/commons-codec/commons-codec/1.10/commons-codec-1.10.jar:/home/hadoop/repository/joda-time/joda-time/2.9.3/joda-time-2.9.3.jar:/home/hadoop/repository/org/jodd/jodd-core/3.5.2/jodd-core-3.5.2.jar:/home/hadoop/repository/org/datanucleus/datanucleus-core/3.2.10/datanucleus-core-3.2.10.jar:/home/hadoop/repository/org/apache/thrift/libthrift/0.9.3/libthrift-0.9.3.jar:/home/hadoop/repository/org/apache/thrift/libfb303/0.9.3/libfb303-0.9.3.jar:/home/hadoop/repository/org/apache/derby/derby/10.12.1.1/derby-10.12.1.1.jar:/home/hadoop/repository/org/apache/spark/spark-streaming_2.11/2.4.1/spark-streaming_2.11-2.4.1.jar:/home/hadoop/repository/org/apache/spark/spark-streaming-kafka-0-10_2.11/2.4.1/spark-streaming-kafka-0-10_2.11-2.4.1.jar:/home/hadoop/repository/org/apache/kafka/kafka-clients/2.0.0/kafka-clients-2.0.0.jar:/home/hadoop/repository/com/alibaba/fastjson/1.2.40/fastjson-1.2.40.jar:/home/hadoop/repository/org/apache/hive/hive-jdbc/2.3.3/hive-jdbc-2.3.3.jar:/home/hadoop/repository/org/apache/hive/hive-common/2.3.3/hive-common-2.3.3.jar:/home/hadoop/repository/org/apache/hive/hive-storage-api/2.4.0/hive-storage-api-2.4.0.jar:/home/hadoop/repository/org/apache/orc/orc-core/1.3.3/orc-core-1.3.3.jar:/home/hadoop/repository/org/eclipse/jetty/aggregate/jetty-all/7.6.0.v20120127/jetty-all-7.6.0.v20120127.jar:/home/hadoop/repository/org/apache/geronimo/specs/geronimo-jta_1.1_spec/1.1.1/geronimo-jta_1.1_spec-1.1.1.jar:/home/hadoop/repository/javax/mail/mail/1.4.1/mail-1.4.1.jar:/home/hadoop/repository/org/apache/geronimo/specs/geronimo-jaspic_1.0_spec/1.0/geronimo-jaspic_1.0_spec-1.0.jar:/home/hadoop/repository/org/apache/geronimo/specs/geronimo-annotation_1.0_spec/1.1.1/geronimo-annotation_1.0_spec-1.1.1.jar:/home/hadoop/repository/asm/asm-commons/3.1/asm-commons-3.1.jar:/home/hadoop/repository/asm/asm-tree/3.1/asm-tree-3.1.jar:/home/hadoop/repository/asm/asm/3.1/asm-3.1.jar:/home/hadoop/repository/org/eclipse/jetty/orbit/javax.servlet/3.0.0.v201112011016/javax.servlet-3.0.0.v201112011016.jar:/home/hadoop/repository/org/apache/logging/log4j/log4j-web/2.6.2/log4j-web-2.6.2.jar:/home/hadoop/repository/com/tdunning/json/1.8/json-1.8.jar:/home/hadoop/repository/com/github/joshelser/dropwizard-metrics-hadoop-metrics2-reporter/0.1.2/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/home/hadoop/repository/org/apache/hive/hive-service/2.3.3/hive-service-2.3.3.jar:/home/hadoop/repository/org/apache/hive/hive-llap-server/2.3.3/hive-llap-server-2.3.3.jar:/home/hadoop/repository/org/apache/hive/hive-llap-common/2.3.3/hive-llap-common-2.3.3.jar:/home/hadoop/repository/org/apache/slider/slider-core/0.90.2-incubating/slider-core-0.90.2-incubating.jar:/home/hadoop/repository/com/beust/jcommander/1.30/jcommander-1.30.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-yarn-registry/2.7.1/hadoop-yarn-registry-2.7.1.jar:/home/hadoop/repository/com/sun/jersey/jersey-client/1.9/jersey-client-1.9.jar:/home/hadoop/repository/com/sun/jersey/jersey-json/1.9/jersey-json-1.9.jar:/home/hadoop/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/home/hadoop/repository/com/sun/jersey/jersey-server/1.9/jersey-server-1.9.jar:/home/hadoop/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/home/hadoop/repository/org/apache/hive/hive-llap-common/2.3.3/hive-llap-common-2.3.3-tests.jar:/home/hadoop/repository/org/apache/hbase/hbase-hadoop2-compat/1.1.1/hbase-hadoop2-compat-1.1.1.jar:/home/hadoop/repository/org/apache/commons/commons-math/2.2/commons-math-2.2.jar:/home/hadoop/repository/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.jar:/home/hadoop/repository/org/apache/hbase/hbase-server/1.1.1/hbase-server-1.1.1.jar:/home/hadoop/repository/org/apache/hbase/hbase-procedure/1.1.1/hbase-procedure-1.1.1.jar:/home/hadoop/repository/org/apache/hbase/hbase-common/1.1.1/hbase-common-1.1.1-tests.jar:/home/hadoop/repository/org/apache/hbase/hbase-prefix-tree/1.1.1/hbase-prefix-tree-1.1.1.jar:/home/hadoop/repository/com/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar:/home/hadoop/repository/org/mortbay/jetty/jetty-sslengine/6.1.26/jetty-sslengine-6.1.26.jar:/home/hadoop/repository/org/mortbay/jetty/jsp-2.1/6.1.14/jsp-2.1-6.1.14.jar:/home/hadoop/repository/org/mortbay/jetty/jsp-api-2.1/6.1.14/jsp-api-2.1-6.1.14.jar:/home/hadoop/repository/org/mortbay/jetty/servlet-api-2.5/6.1.14/servlet-api-2.5-6.1.14.jar:/home/hadoop/repository/com/lmax/disruptor/3.3.0/disruptor-3.3.0.jar:/home/hadoop/repository/org/apache/hbase/hbase-common/1.1.1/hbase-common-1.1.1.jar:/home/hadoop/repository/org/apache/hbase/hbase-hadoop-compat/1.1.1/hbase-hadoop-compat-1.1.1.jar:/home/hadoop/repository/net/sf/jpam/jpam/1.1/jpam-1.1.jar:/home/hadoop/repository/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar:/home/hadoop/repository/javax/servlet/jsp-api/2.0/jsp-api-2.0.jar:/home/hadoop/repository/ant/ant/1.6.5/ant-1.6.5.jar:/home/hadoop/repository/tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar:/home/hadoop/repository/javax/servlet/servlet-api/2.4/servlet-api-2.4.jar:/home/hadoop/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/home/hadoop/repository/org/jamon/jamon-runtime/2.3.1/jamon-runtime-2.3.1.jar:/home/hadoop/repository/org/apache/hive/hive-serde/2.3.3/hive-serde-2.3.3.jar:/home/hadoop/repository/org/apache/parquet/parquet-hadoop-bundle/1.8.1/parquet-hadoop-bundle-1.8.1.jar:/home/hadoop/repository/org/apache/hive/hive-metastore/2.3.3/hive-metastore-2.3.3.jar:/home/hadoop/repository/org/apache/hbase/hbase-client/1.1.1/hbase-client-1.1.1.jar:/home/hadoop/repository/org/apache/hbase/hbase-annotations/1.1.1/hbase-annotations-1.1.1.jar:/usr/lib/jvm/java-1.8.0-openjdk/lib/tools.jar:/home/hadoop/repository/org/apache/hbase/hbase-protocol/1.1.1/hbase-protocol-1.1.1.jar:/home/hadoop/repository/org/jruby/jcodings/jcodings/1.0.8/jcodings-1.0.8.jar:/home/hadoop/repository/org/jruby/joni/joni/2.1.2/joni-2.1.2.jar:/home/hadoop/repository/com/github/stephenc/findbugs/findbugs-annotations/1.3.9-1/findbugs-annotations-1.3.9-1.jar:/home/hadoop/repository/com/zaxxer/HikariCP/2.5.1/HikariCP-2.5.1.jar:/home/hadoop/repository/org/datanucleus/javax.jdo/3.2.0-m3/javax.jdo-3.2.0-m3.jar:/home/hadoop/repository/javax/transaction/transaction-api/1.1/transaction-api-1.1.jar:/home/hadoop/repository/co/cask/tephra/tephra-api/0.6.0/tephra-api-0.6.0.jar:/home/hadoop/repository/co/cask/tephra/tephra-core/0.6.0/tephra-core-0.6.0.jar:/home/hadoop/repository/com/google/inject/extensions/guice-assistedinject/3.0/guice-assistedinject-3.0.jar:/home/hadoop/repository/it/unimi/dsi/fastutil/6.5.6/fastutil-6.5.6.jar:/home/hadoop/repository/org/apache/twill/twill-common/0.6.0-incubating/twill-common-0.6.0-incubating.jar:/home/hadoop/repository/org/apache/twill/twill-core/0.6.0-incubating/twill-core-0.6.0-incubating.jar:/home/hadoop/repository/org/apache/twill/twill-api/0.6.0-incubating/twill-api-0.6.0-incubating.jar:/home/hadoop/repository/org/apache/twill/twill-discovery-api/0.6.0-incubating/twill-discovery-api-0.6.0-incubating.jar:/home/hadoop/repository/org/apache/twill/twill-discovery-core/0.6.0-incubating/twill-discovery-core-0.6.0-incubating.jar:/home/hadoop/repository/org/apache/twill/twill-zookeeper/0.6.0-incubating/twill-zookeeper-0.6.0-incubating.jar:/home/hadoop/repository/co/cask/tephra/tephra-hbase-compat-1.0/0.6.0/tephra-hbase-compat-1.0-0.6.0.jar:/home/hadoop/repository/org/apache/hive/hive-shims/2.3.3/hive-shims-2.3.3.jar:/home/hadoop/repository/org/apache/hive/shims/hive-shims-common/2.3.3/hive-shims-common-2.3.3.jar:/home/hadoop/repository/org/apache/hive/shims/hive-shims-0.23/2.3.3/hive-shims-0.23-2.3.3.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.2/hadoop-yarn-server-resourcemanager-2.7.2.jar:/home/hadoop/repository/com/sun/jersey/contribs/jersey-guice/1.9/jersey-guice-1.9.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.2/hadoop-yarn-server-applicationhistoryservice-2.7.2.jar:/home/hadoop/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar:/home/hadoop/repository/org/apache/hive/shims/hive-shims-scheduler/2.3.3/hive-shims-scheduler-2.3.3.jar:/home/hadoop/repository/org/apache/hive/hive-service-rpc/2.3.3/hive-service-rpc-2.3.3.jar:/home/hadoop/repository/org/apache/httpcomponents/httpcore/4.4/httpcore-4.4.jar:/home/hadoop/repository/org/apache/curator/curator-framework/2.7.1/curator-framework-2.7.1.jar:/home/hadoop/repository/org/apache/curator/curator-client/2.7.1/curator-client-2.7.1.jar:/home/hadoop/repository/org/apache/hive/hive-exec/2.3.3/hive-exec-2.3.3.jar:/home/hadoop/repository/org/apache/hive/hive-vector-code-gen/2.3.3/hive-vector-code-gen-2.3.3.jar:/home/hadoop/repository/org/apache/velocity/velocity/1.5/velocity-1.5.jar:/home/hadoop/repository/org/apache/hive/hive-llap-tez/2.3.3/hive-llap-tez-2.3.3.jar:/home/hadoop/repository/org/apache/hive/hive-llap-client/2.3.3/hive-llap-client-2.3.3.jar:/home/hadoop/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/home/hadoop/repository/org/apache/logging/log4j/log4j-1.2-api/2.6.2/log4j-1.2-api-2.6.2.jar:/home/hadoop/repository/org/apache/logging/log4j/log4j-api/2.6.2/log4j-api-2.6.2.jar:/home/hadoop/repository/org/apache/logging/log4j/log4j-core/2.6.2/log4j-core-2.6.2.jar:/home/hadoop/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.6.2/log4j-slf4j-impl-2.6.2.jar:/home/hadoop/repository/org/antlr/antlr-runtime/3.5.2/antlr-runtime-3.5.2.jar:/home/hadoop/repository/org/antlr/ST4/4.0.4/ST4-4.0.4.jar:/home/hadoop/repository/org/apache/ant/ant/1.9.1/ant-1.9.1.jar:/home/hadoop/repository/org/apache/ant/ant-launcher/1.9.1/ant-launcher-1.9.1.jar:/home/hadoop/repository/org/apache/commons/commons-compress/1.9/commons-compress-1.9.jar:/home/hadoop/repository/org/codehaus/groovy/groovy-all/2.4.4/groovy-all-2.4.4.jar:/home/hadoop/repository/org/apache/calcite/calcite-druid/1.10.0/calcite-druid-1.10.0.jar:/home/hadoop/repository/org/apache/calcite/avatica/avatica/1.8.0/avatica-1.8.0.jar:/home/hadoop/repository/org/apache/calcite/avatica/avatica-metrics/1.8.0/avatica-metrics-1.8.0.jar:/home/hadoop/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/home/hadoop/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-client/2.7.3/hadoop-client-2.7.3.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3.jar:/home/hadoop/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/home/hadoop/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/home/hadoop/repository/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar:/home/hadoop/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/home/hadoop/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/home/hadoop/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/home/hadoop/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-auth/2.7.3/hadoop-auth-2.7.3.jar:/home/hadoop/repository/org/apache/directory/server/apacheds-kerberos-codec/2.0.0-M15/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadoop/repository/org/apache/directory/server/apacheds-i18n/2.0.0-M15/apacheds-i18n-2.0.0-M15.jar:/home/hadoop/repository/org/apache/directory/api/api-asn1-api/1.0.0-M20/api-asn1-api-1.0.0-M20.jar:/home/hadoop/repository/org/apache/directory/api/api-util/1.0.0-M20/api-util-1.0.0-M20.jar:/home/hadoop/repository/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3.jar:/home/hadoop/repository/xerces/xercesImpl/2.9.1/xercesImpl-2.9.1.jar:/home/hadoop/repository/xml-apis/xml-apis/1.3.04/xml-apis-1.3.04.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-mapreduce-client-app/2.7.3/hadoop-mapreduce-client-app-2.7.3.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.7.3/hadoop-mapreduce-client-common-2.7.3.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-mapreduce-client-shuffle/2.7.3/hadoop-mapreduce-client-shuffle-2.7.3.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.3/hadoop-mapreduce-client-core-2.7.3.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.3/hadoop-mapreduce-client-jobclient-2.7.3.jar:/home/hadoop/repository/org/apache/hadoop/hadoop-annotations/2.7.3/hadoop-annotations-2.7.3.jar:/home/hadoop/repository/mysql/mysql-connector-java/5.1.35/mysql-connector-java-5.1.35.jar:/home/hadoop/repository/com/google/guava/guava/26.0-jre/guava-26.0-jre.jar:/home/hadoop/repository/org/checkerframework/checker-qual/2.5.2/checker-qual-2.5.2.jar:/home/hadoop/repository/com/google/errorprone/error_prone_annotations/2.1.3/error_prone_annotations-2.1.3.jar:/home/hadoop/repository/com/google/j2objc/j2objc-annotations/1.1/j2objc-annotations-1.1.jar:/home/hadoop/repository/org/codehaus/mojo/animal-sniffer-annotations/1.14/animal-sniffer-annotations-1.14.jar:/home/hadoop/scala/lib/scala-parser-combinators_2.11-1.0.4.jar:/home/hadoop/scala/lib/scala-reflect.jar:/home/hadoop/scala/lib/scala-actors-migration_2.11-1.1.0.jar:/home/hadoop/scala/lib/scala-xml_2.11-1.0.4.jar:/home/hadoop/scala/lib/scala-library.jar:/home/hadoop/scala/lib/scala-swing_2.11-1.0.2.jar:/home/hadoop/scala/lib/scala-actors-2.11.0.jar:/home/hadoop/idea/lib/idea_rt.jar com.xh.spark.sql.function.WindowFunctionTest
Connected to the target VM, address: '127.0.0.1:45315', transport: 'socket'
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.6.2/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/04/12 16:47:02 INFO SparkContext: Running Spark version 2.4.1
19/04/12 16:47:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/04/12 16:47:04 INFO SparkContext: Submitted application: WindowFunctionTest
19/04/12 16:47:05 INFO SecurityManager: Changing view acls to: hadoop
19/04/12 16:47:05 INFO SecurityManager: Changing modify acls to: hadoop
19/04/12 16:47:05 INFO SecurityManager: Changing view acls groups to: 
19/04/12 16:47:05 INFO SecurityManager: Changing modify acls groups to: 
19/04/12 16:47:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set(); users  with modify permissions: Set(hadoop); groups with modify permissions: Set()
19/04/12 16:47:07 INFO Utils: Successfully started service 'sparkDriver' on port 43568.
19/04/12 16:47:07 INFO SparkEnv: Registering MapOutputTracker
19/04/12 16:47:07 INFO SparkEnv: Registering BlockManagerMaster
19/04/12 16:47:07 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/04/12 16:47:07 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/04/12 16:47:07 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-07913749-f493-4bae-95aa-f9ea019dde51
19/04/12 16:47:07 INFO MemoryStore: MemoryStore started with capacity 447.3 MB
19/04/12 16:47:07 INFO SparkEnv: Registering OutputCommitCoordinator
19/04/12 16:47:08 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/04/12 16:47:08 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://hadoop04:4040
19/04/12 16:47:08 INFO SparkContext: Added JAR /home/hadoop/worker/sparklearning/target/sparklearning-1.0-SNAPSHOT.jar at spark://hadoop04:43568/jars/sparklearning-1.0-SNAPSHOT.jar with timestamp 1555058828687
19/04/12 16:47:08 INFO SparkContext: Added JAR /home/hadoop/worker/sparklearning/target/sparklearning-1.0-SNAPSHOT-jar-with-dependencies.jar at spark://hadoop04:43568/jars/sparklearning-1.0-SNAPSHOT-jar-with-dependencies.jar with timestamp 1555058828703
19/04/12 16:47:13 INFO Client: Requesting a new application from cluster with 3 NodeManagers
19/04/12 16:47:13 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
19/04/12 16:47:13 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
19/04/12 16:47:13 INFO Client: Setting up container launch context for our AM
19/04/12 16:47:13 INFO Client: Setting up the launch environment for our AM container
19/04/12 16:47:14 INFO Client: Preparing resources for our AM container
19/04/12 16:47:14 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
19/04/12 16:47:23 INFO Client: Uploading resource file:/tmp/spark-f3533ae6-b45b-4a1f-8119-0c12775c5a33/__spark_libs__8176900855339303501.zip -> hdfs://ns1/user/hadoop/.sparkStaging/application_1555049463744_0005/__spark_libs__8176900855339303501.zip
19/04/12 16:47:50 INFO Client: Uploading resource file:/tmp/spark-f3533ae6-b45b-4a1f-8119-0c12775c5a33/__spark_conf__4747135480716905311.zip -> hdfs://ns1/user/hadoop/.sparkStaging/application_1555049463744_0005/__spark_conf__.zip
19/04/12 16:47:51 INFO SecurityManager: Changing view acls to: hadoop
19/04/12 16:47:51 INFO SecurityManager: Changing modify acls to: hadoop
19/04/12 16:47:51 INFO SecurityManager: Changing view acls groups to: 
19/04/12 16:47:51 INFO SecurityManager: Changing modify acls groups to: 
19/04/12 16:47:51 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set(); users  with modify permissions: Set(hadoop); groups with modify permissions: Set()
19/04/12 16:47:55 INFO Client: Submitting application application_1555049463744_0005 to ResourceManager
19/04/12 16:47:55 INFO YarnClientImpl: Submitted application application_1555049463744_0005
19/04/12 16:47:55 INFO SchedulerExtensionServices: Starting Yarn extension services with app application_1555049463744_0005 and attemptId None
19/04/12 16:47:57 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:47:57 INFO Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: root.hadoop
	 start time: 1555058875812
	 final status: UNDEFINED
	 tracking URL: http://192.168.8.82:8089/proxy/application_1555049463744_0005/
	 user: hadoop
19/04/12 16:47:58 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:47:59 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:00 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:01 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:02 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:03 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:04 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:05 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:06 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:07 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:08 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:09 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:10 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:11 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:12 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:13 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:14 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:15 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:16 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:17 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:18 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:19 INFO Client: Application report for application_1555049463744_0005 (state: ACCEPTED)
19/04/12 16:48:20 INFO Client: Application report for application_1555049463744_0005 (state: RUNNING)
19/04/12 16:48:20 INFO Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: 192.168.8.82
	 ApplicationMaster RPC port: -1
	 queue: root.hadoop
	 start time: 1555058875812
	 final status: UNDEFINED
	 tracking URL: http://192.168.8.82:8089/proxy/application_1555049463744_0005/
	 user: hadoop
19/04/12 16:48:20 INFO YarnClientSchedulerBackend: Application application_1555049463744_0005 has started running.
19/04/12 16:48:21 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 42715.
19/04/12 16:48:21 INFO NettyBlockTransferService: Server created on hadoop04:42715
19/04/12 16:48:21 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/04/12 16:48:21 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, hadoop04, 42715, None)
19/04/12 16:48:21 INFO BlockManagerMasterEndpoint: Registering block manager hadoop04:42715 with 447.3 MB RAM, BlockManagerId(driver, hadoop04, 42715, None)
19/04/12 16:48:21 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, hadoop04, 42715, None)
19/04/12 16:48:22 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, hadoop04, 42715, None)
19/04/12 16:48:23 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms)
19/04/12 16:48:24 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> 192.168.8.82, PROXY_URI_BASES -> http://192.168.8.82:8089/proxy/application_1555049463744_0005, RM_HA_URLS -> hadoop02:8088,hadoop03:8088), /proxy/application_1555049463744_0005
19/04/12 16:48:24 INFO JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /jobs, /jobs/json, /jobs/job, /jobs/job/json, /stages, /stages/json, /stages/stage, /stages/stage/json, /stages/pool, /stages/pool/json, /storage, /storage/json, /storage/rdd, /storage/rdd/json, /environment, /environment/json, /executors, /executors/json, /executors/threadDump, /executors/threadDump/json, /static, /, /api, /jobs/job/kill, /stages/stage/kill, /metrics/json.
19/04/12 16:48:24 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
19/04/12 16:48:32 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/home/hadoop/worker/sparklearning/spark-warehouse').
19/04/12 16:48:32 INFO SharedState: Warehouse path is 'file:/home/hadoop/worker/sparklearning/spark-warehouse'.
19/04/12 16:48:32 INFO JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL.
19/04/12 16:48:32 INFO JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/json.
19/04/12 16:48:32 INFO JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution.
19/04/12 16:48:32 INFO JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution/json.
19/04/12 16:48:32 INFO JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /static/sql.
19/04/12 16:48:52 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
19/04/12 16:48:55 INFO CodeGenerator: Code generated in 2751.822246 ms
19/04/12 16:52:26 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.8.83:53042) with ID 1
19/04/12 16:52:30 INFO BlockManagerMasterEndpoint: Registering block manager hadoop03:45104 with 366.3 MB RAM, BlockManagerId(1, hadoop03, 45104, None)
19/04/12 16:52:35 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.8.84:37146) with ID 2
19/04/12 16:52:39 INFO BlockManagerMasterEndpoint: Registering block manager hadoop04:38267 with 366.3 MB RAM, BlockManagerId(2, hadoop04, 38267, None)
19/04/12 16:53:38 INFO CodeGenerator: Code generated in 668.087005 ms
19/04/12 16:53:45 INFO CodeGenerator: Code generated in 732.99189 ms
19/04/12 16:53:47 INFO CodeGenerator: Code generated in 1265.744172 ms
19/04/12 16:53:47 INFO CodeGenerator: Code generated in 271.771791 ms
19/04/12 16:53:51 INFO SparkContext: Starting job: show at WindowFunctionTest.scala:45
19/04/12 16:53:51 INFO DAGScheduler: Registering RDD 2 (show at WindowFunctionTest.scala:45)
19/04/12 16:53:51 INFO DAGScheduler: Got job 0 (show at WindowFunctionTest.scala:45) with 1 output partitions
19/04/12 16:53:51 INFO DAGScheduler: Final stage: ResultStage 1 (show at WindowFunctionTest.scala:45)
19/04/12 16:53:51 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
19/04/12 16:53:51 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
19/04/12 16:53:51 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[2] at show at WindowFunctionTest.scala:45), which has no missing parents
19/04/12 16:53:54 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 5.1 KB, free 447.3 MB)
19/04/12 16:53:55 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.1 KB, free 447.3 MB)
19/04/12 16:53:55 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on hadoop04:42715 (size: 3.1 KB, free: 447.3 MB)
19/04/12 16:53:55 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1161
19/04/12 16:53:55 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[2] at show at WindowFunctionTest.scala:45) (first 15 tasks are for partitions Vector(0, 1))
19/04/12 16:53:55 INFO YarnScheduler: Adding task set 0.0 with 2 tasks
19/04/12 16:53:56 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, hadoop03, executor 1, partition 0, PROCESS_LOCAL, 8230 bytes)
19/04/12 16:53:56 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, hadoop04, executor 2, partition 1, PROCESS_LOCAL, 8230 bytes)
19/04/12 16:55:52 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on hadoop03:45104 (size: 3.1 KB, free: 366.3 MB)
19/04/12 16:55:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on hadoop04:38267 (size: 3.1 KB, free: 366.3 MB)
19/04/12 16:56:13 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 137230 ms on hadoop03 (executor 1) (1/2)
19/04/12 16:56:24 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 147247 ms on hadoop04 (executor 2) (2/2)
19/04/12 16:56:24 INFO YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool 
19/04/12 16:56:24 INFO DAGScheduler: ShuffleMapStage 0 (show at WindowFunctionTest.scala:45) finished in 151.743 s
19/04/12 16:56:24 INFO DAGScheduler: looking for newly runnable stages
19/04/12 16:56:24 INFO DAGScheduler: running: Set()
19/04/12 16:56:24 INFO DAGScheduler: waiting: Set(ResultStage 1)
19/04/12 16:56:24 INFO DAGScheduler: failed: Set()
19/04/12 16:56:24 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[8] at show at WindowFunctionTest.scala:45), which has no missing parents
19/04/12 16:56:24 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 18.4 KB, free 447.3 MB)
19/04/12 16:56:25 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 8.8 KB, free 447.3 MB)
19/04/12 16:56:25 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on hadoop04:42715 (size: 8.8 KB, free: 447.3 MB)
19/04/12 16:56:25 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1161
19/04/12 16:56:25 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[8] at show at WindowFunctionTest.scala:45) (first 15 tasks are for partitions Vector(0))
19/04/12 16:56:25 INFO YarnScheduler: Adding task set 1.0 with 1 tasks
19/04/12 16:56:25 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, hadoop03, executor 1, partition 0, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:25 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on hadoop03:45104 (size: 8.8 KB, free: 366.3 MB)
19/04/12 16:56:26 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.168.8.83:53042
19/04/12 16:56:27 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 2345 ms on hadoop03 (executor 1) (1/1)
19/04/12 16:56:27 INFO YarnScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool 
19/04/12 16:56:27 INFO DAGScheduler: ResultStage 1 (show at WindowFunctionTest.scala:45) finished in 3.127 s
19/04/12 16:56:27 INFO DAGScheduler: Job 0 finished: show at WindowFunctionTest.scala:45, took 156.360095 s
19/04/12 16:56:28 INFO SparkContext: Starting job: show at WindowFunctionTest.scala:45
19/04/12 16:56:28 INFO DAGScheduler: Got job 1 (show at WindowFunctionTest.scala:45) with 4 output partitions
19/04/12 16:56:28 INFO DAGScheduler: Final stage: ResultStage 3 (show at WindowFunctionTest.scala:45)
19/04/12 16:56:28 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 2)
19/04/12 16:56:28 INFO DAGScheduler: Missing parents: List()
19/04/12 16:56:28 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[8] at show at WindowFunctionTest.scala:45), which has no missing parents
19/04/12 16:56:28 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 18.4 KB, free 447.2 MB)
19/04/12 16:56:28 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 8.8 KB, free 447.2 MB)
19/04/12 16:56:28 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on hadoop04:42715 (size: 8.8 KB, free: 447.3 MB)
19/04/12 16:56:28 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1161
19/04/12 16:56:28 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 3 (MapPartitionsRDD[8] at show at WindowFunctionTest.scala:45) (first 15 tasks are for partitions Vector(1, 2, 3, 4))
19/04/12 16:56:28 INFO YarnScheduler: Adding task set 3.0 with 4 tasks
19/04/12 16:56:28 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, hadoop04, executor 2, partition 1, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:28 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 4, hadoop03, executor 1, partition 2, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:28 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on hadoop03:45104 (size: 8.8 KB, free: 366.3 MB)
19/04/12 16:56:29 INFO TaskSetManager: Starting task 2.0 in stage 3.0 (TID 5, hadoop03, executor 1, partition 3, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:29 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 4) in 447 ms on hadoop03 (executor 1) (1/4)
19/04/12 16:56:29 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on hadoop04:38267 (size: 8.8 KB, free: 366.3 MB)
19/04/12 16:56:29 INFO TaskSetManager: Starting task 3.0 in stage 3.0 (TID 6, hadoop03, executor 1, partition 4, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:29 INFO TaskSetManager: Finished task 2.0 in stage 3.0 (TID 5) in 290 ms on hadoop03 (executor 1) (2/4)
19/04/12 16:56:29 INFO TaskSetManager: Finished task 3.0 in stage 3.0 (TID 6) in 334 ms on hadoop03 (executor 1) (3/4)
19/04/12 16:56:30 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.168.8.84:37146
19/04/12 16:56:31 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 3241 ms on hadoop04 (executor 2) (4/4)
19/04/12 16:56:31 INFO YarnScheduler: Removed TaskSet 3.0, whose tasks have all completed, from pool 
19/04/12 16:56:31 INFO DAGScheduler: ResultStage 3 (show at WindowFunctionTest.scala:45) finished in 3.803 s
19/04/12 16:56:31 INFO DAGScheduler: Job 1 finished: show at WindowFunctionTest.scala:45, took 3.848347 s
19/04/12 16:56:32 INFO SparkContext: Starting job: show at WindowFunctionTest.scala:45
19/04/12 16:56:32 INFO DAGScheduler: Got job 2 (show at WindowFunctionTest.scala:45) with 20 output partitions
19/04/12 16:56:32 INFO DAGScheduler: Final stage: ResultStage 5 (show at WindowFunctionTest.scala:45)
19/04/12 16:56:32 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 4)
19/04/12 16:56:32 INFO DAGScheduler: Missing parents: List()
19/04/12 16:56:32 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[8] at show at WindowFunctionTest.scala:45), which has no missing parents
19/04/12 16:56:32 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 18.4 KB, free 447.2 MB)
19/04/12 16:56:32 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 8.8 KB, free 447.2 MB)
19/04/12 16:56:32 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on hadoop04:42715 (size: 8.8 KB, free: 447.3 MB)
19/04/12 16:56:32 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1161
19/04/12 16:56:32 INFO DAGScheduler: Submitting 20 missing tasks from ResultStage 5 (MapPartitionsRDD[8] at show at WindowFunctionTest.scala:45) (first 15 tasks are for partitions Vector(5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19))
19/04/12 16:56:32 INFO YarnScheduler: Adding task set 5.0 with 20 tasks
19/04/12 16:56:32 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 7, hadoop03, executor 1, partition 5, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:32 INFO TaskSetManager: Starting task 1.0 in stage 5.0 (TID 8, hadoop04, executor 2, partition 6, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:32 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on hadoop04:38267 (size: 8.8 KB, free: 366.3 MB)
19/04/12 16:56:32 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on hadoop03:45104 (size: 8.8 KB, free: 366.3 MB)
19/04/12 16:56:33 INFO TaskSetManager: Starting task 2.0 in stage 5.0 (TID 9, hadoop04, executor 2, partition 7, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:33 INFO TaskSetManager: Finished task 1.0 in stage 5.0 (TID 8) in 726 ms on hadoop04 (executor 2) (1/20)
19/04/12 16:56:33 INFO TaskSetManager: Starting task 3.0 in stage 5.0 (TID 10, hadoop03, executor 1, partition 8, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:33 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 7) in 811 ms on hadoop03 (executor 1) (2/20)
19/04/12 16:56:33 INFO TaskSetManager: Starting task 4.0 in stage 5.0 (TID 11, hadoop03, executor 1, partition 9, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:33 INFO TaskSetManager: Finished task 3.0 in stage 5.0 (TID 10) in 393 ms on hadoop03 (executor 1) (3/20)
19/04/12 16:56:33 INFO TaskSetManager: Starting task 5.0 in stage 5.0 (TID 12, hadoop04, executor 2, partition 10, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:33 INFO TaskSetManager: Finished task 2.0 in stage 5.0 (TID 9) in 523 ms on hadoop04 (executor 2) (4/20)
19/04/12 16:56:33 INFO TaskSetManager: Starting task 6.0 in stage 5.0 (TID 13, hadoop03, executor 1, partition 11, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:33 INFO TaskSetManager: Finished task 4.0 in stage 5.0 (TID 11) in 317 ms on hadoop03 (executor 1) (5/20)
19/04/12 16:56:34 INFO TaskSetManager: Starting task 7.0 in stage 5.0 (TID 14, hadoop03, executor 1, partition 12, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:34 INFO TaskSetManager: Starting task 8.0 in stage 5.0 (TID 15, hadoop03, executor 1, partition 13, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:34 INFO TaskSetManager: Finished task 6.0 in stage 5.0 (TID 13) in 851 ms on hadoop03 (executor 1) (6/20)
19/04/12 16:56:34 INFO TaskSetManager: Finished task 7.0 in stage 5.0 (TID 14) in 541 ms on hadoop03 (executor 1) (7/20)
19/04/12 16:56:34 INFO TaskSetManager: Starting task 9.0 in stage 5.0 (TID 16, hadoop04, executor 2, partition 14, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:34 INFO TaskSetManager: Finished task 5.0 in stage 5.0 (TID 12) in 1219 ms on hadoop04 (executor 2) (8/20)
19/04/12 16:56:34 INFO TaskSetManager: Starting task 10.0 in stage 5.0 (TID 17, hadoop04, executor 2, partition 15, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:34 INFO TaskSetManager: Starting task 11.0 in stage 5.0 (TID 18, hadoop03, executor 1, partition 16, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:34 INFO TaskSetManager: Finished task 9.0 in stage 5.0 (TID 16) in 313 ms on hadoop04 (executor 2) (9/20)
19/04/12 16:56:34 INFO TaskSetManager: Finished task 8.0 in stage 5.0 (TID 15) in 596 ms on hadoop03 (executor 1) (10/20)
19/04/12 16:56:35 INFO TaskSetManager: Starting task 12.0 in stage 5.0 (TID 19, hadoop04, executor 2, partition 17, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:35 INFO TaskSetManager: Starting task 13.0 in stage 5.0 (TID 20, hadoop03, executor 1, partition 18, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:35 INFO TaskSetManager: Finished task 11.0 in stage 5.0 (TID 18) in 158 ms on hadoop03 (executor 1) (11/20)
19/04/12 16:56:35 INFO TaskSetManager: Finished task 10.0 in stage 5.0 (TID 17) in 167 ms on hadoop04 (executor 2) (12/20)
19/04/12 16:56:35 INFO TaskSetManager: Starting task 14.0 in stage 5.0 (TID 21, hadoop03, executor 1, partition 19, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:35 INFO TaskSetManager: Starting task 15.0 in stage 5.0 (TID 22, hadoop04, executor 2, partition 20, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:35 INFO TaskSetManager: Finished task 13.0 in stage 5.0 (TID 20) in 250 ms on hadoop03 (executor 1) (13/20)
19/04/12 16:56:35 INFO TaskSetManager: Finished task 12.0 in stage 5.0 (TID 19) in 343 ms on hadoop04 (executor 2) (14/20)
19/04/12 16:56:35 INFO TaskSetManager: Starting task 16.0 in stage 5.0 (TID 23, hadoop04, executor 2, partition 21, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:35 INFO TaskSetManager: Finished task 15.0 in stage 5.0 (TID 22) in 247 ms on hadoop04 (executor 2) (15/20)
19/04/12 16:56:35 INFO TaskSetManager: Starting task 17.0 in stage 5.0 (TID 24, hadoop03, executor 1, partition 22, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:35 INFO TaskSetManager: Finished task 14.0 in stage 5.0 (TID 21) in 481 ms on hadoop03 (executor 1) (16/20)
19/04/12 16:56:35 INFO TaskSetManager: Starting task 18.0 in stage 5.0 (TID 25, hadoop04, executor 2, partition 23, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:35 INFO TaskSetManager: Finished task 16.0 in stage 5.0 (TID 23) in 393 ms on hadoop04 (executor 2) (17/20)
19/04/12 16:56:36 INFO TaskSetManager: Starting task 19.0 in stage 5.0 (TID 26, hadoop04, executor 2, partition 24, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:36 INFO TaskSetManager: Finished task 18.0 in stage 5.0 (TID 25) in 473 ms on hadoop04 (executor 2) (18/20)
19/04/12 16:56:36 INFO TaskSetManager: Finished task 17.0 in stage 5.0 (TID 24) in 898 ms on hadoop03 (executor 1) (19/20)
19/04/12 16:56:36 INFO TaskSetManager: Finished task 19.0 in stage 5.0 (TID 26) in 524 ms on hadoop04 (executor 2) (20/20)
19/04/12 16:56:36 INFO YarnScheduler: Removed TaskSet 5.0, whose tasks have all completed, from pool 
19/04/12 16:56:36 INFO DAGScheduler: ResultStage 5 (show at WindowFunctionTest.scala:45) finished in 4.493 s
19/04/12 16:56:36 INFO DAGScheduler: Job 2 finished: show at WindowFunctionTest.scala:45, took 4.566402 s
19/04/12 16:56:37 INFO SparkContext: Starting job: show at WindowFunctionTest.scala:45
19/04/12 16:56:37 INFO DAGScheduler: Got job 3 (show at WindowFunctionTest.scala:45) with 100 output partitions
19/04/12 16:56:37 INFO DAGScheduler: Final stage: ResultStage 7 (show at WindowFunctionTest.scala:45)
19/04/12 16:56:37 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 6)
19/04/12 16:56:37 INFO DAGScheduler: Missing parents: List()
19/04/12 16:56:37 INFO DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[8] at show at WindowFunctionTest.scala:45), which has no missing parents
19/04/12 16:56:37 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 18.4 KB, free 447.2 MB)
19/04/12 16:56:37 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 8.8 KB, free 447.2 MB)
19/04/12 16:56:37 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on hadoop04:42715 (size: 8.8 KB, free: 447.3 MB)
19/04/12 16:56:37 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1161
19/04/12 16:56:37 INFO DAGScheduler: Submitting 100 missing tasks from ResultStage 7 (MapPartitionsRDD[8] at show at WindowFunctionTest.scala:45) (first 15 tasks are for partitions Vector(25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39))
19/04/12 16:56:37 INFO YarnScheduler: Adding task set 7.0 with 100 tasks
19/04/12 16:56:37 INFO TaskSetManager: Starting task 85.0 in stage 7.0 (TID 27, hadoop04, executor 2, partition 110, NODE_LOCAL, 7778 bytes)
19/04/12 16:56:37 INFO TaskSetManager: Starting task 77.0 in stage 7.0 (TID 28, hadoop03, executor 1, partition 102, NODE_LOCAL, 7778 bytes)
19/04/12 16:56:37 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on hadoop04:38267 (size: 8.8 KB, free: 366.3 MB)
19/04/12 16:56:37 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on hadoop03:45104 (size: 8.8 KB, free: 366.3 MB)
19/04/12 16:56:39 INFO TaskSetManager: Starting task 0.0 in stage 7.0 (TID 29, hadoop04, executor 2, partition 25, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:39 INFO TaskSetManager: Finished task 85.0 in stage 7.0 (TID 27) in 1607 ms on hadoop04 (executor 2) (1/100)
19/04/12 16:56:39 INFO TaskSetManager: Starting task 1.0 in stage 7.0 (TID 30, hadoop04, executor 2, partition 26, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:40 INFO TaskSetManager: Starting task 2.0 in stage 7.0 (TID 31, hadoop04, executor 2, partition 27, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:40 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID 29) in 1083 ms on hadoop04 (executor 2) (2/100)
19/04/12 16:56:40 INFO TaskSetManager: Finished task 1.0 in stage 7.0 (TID 30) in 659 ms on hadoop04 (executor 2) (3/100)
19/04/12 16:56:40 INFO TaskSetManager: Starting task 3.0 in stage 7.0 (TID 32, hadoop04, executor 2, partition 28, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:40 INFO TaskSetManager: Finished task 2.0 in stage 7.0 (TID 31) in 473 ms on hadoop04 (executor 2) (4/100)
19/04/12 16:56:40 INFO TaskSetManager: Starting task 4.0 in stage 7.0 (TID 33, hadoop04, executor 2, partition 29, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:41 INFO TaskSetManager: Starting task 5.0 in stage 7.0 (TID 34, hadoop03, executor 1, partition 30, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:41 INFO TaskSetManager: Finished task 77.0 in stage 7.0 (TID 28) in 3578 ms on hadoop03 (executor 1) (5/100)
19/04/12 16:56:41 INFO TaskSetManager: Finished task 3.0 in stage 7.0 (TID 32) in 505 ms on hadoop04 (executor 2) (6/100)
19/04/12 16:56:41 INFO TaskSetManager: Starting task 6.0 in stage 7.0 (TID 35, hadoop04, executor 2, partition 31, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:41 INFO TaskSetManager: Finished task 4.0 in stage 7.0 (TID 33) in 438 ms on hadoop04 (executor 2) (7/100)
19/04/12 16:56:41 INFO TaskSetManager: Starting task 7.0 in stage 7.0 (TID 36, hadoop03, executor 1, partition 32, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:41 INFO TaskSetManager: Starting task 8.0 in stage 7.0 (TID 37, hadoop04, executor 2, partition 33, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:41 INFO TaskSetManager: Finished task 6.0 in stage 7.0 (TID 35) in 353 ms on hadoop04 (executor 2) (8/100)
19/04/12 16:56:41 INFO TaskSetManager: Finished task 5.0 in stage 7.0 (TID 34) in 583 ms on hadoop03 (executor 1) (9/100)
19/04/12 16:56:41 INFO TaskSetManager: Starting task 9.0 in stage 7.0 (TID 38, hadoop03, executor 1, partition 34, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:41 INFO TaskSetManager: Finished task 7.0 in stage 7.0 (TID 36) in 301 ms on hadoop03 (executor 1) (10/100)
19/04/12 16:56:42 INFO TaskSetManager: Starting task 10.0 in stage 7.0 (TID 39, hadoop03, executor 1, partition 35, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:42 INFO TaskSetManager: Starting task 11.0 in stage 7.0 (TID 40, hadoop04, executor 2, partition 36, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:42 INFO TaskSetManager: Finished task 9.0 in stage 7.0 (TID 38) in 453 ms on hadoop03 (executor 1) (11/100)
19/04/12 16:56:42 INFO TaskSetManager: Starting task 12.0 in stage 7.0 (TID 41, hadoop04, executor 2, partition 37, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:42 INFO TaskSetManager: Finished task 8.0 in stage 7.0 (TID 37) in 1017 ms on hadoop04 (executor 2) (12/100)
19/04/12 16:56:42 INFO TaskSetManager: Starting task 13.0 in stage 7.0 (TID 42, hadoop03, executor 1, partition 38, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:42 INFO TaskSetManager: Finished task 10.0 in stage 7.0 (TID 39) in 407 ms on hadoop03 (executor 1) (13/100)
19/04/12 16:56:42 INFO TaskSetManager: Finished task 11.0 in stage 7.0 (TID 40) in 486 ms on hadoop04 (executor 2) (14/100)
19/04/12 16:56:42 INFO TaskSetManager: Starting task 14.0 in stage 7.0 (TID 43, hadoop04, executor 2, partition 39, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:42 INFO TaskSetManager: Finished task 12.0 in stage 7.0 (TID 41) in 259 ms on hadoop04 (executor 2) (15/100)
19/04/12 16:56:43 INFO TaskSetManager: Starting task 15.0 in stage 7.0 (TID 44, hadoop04, executor 2, partition 40, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:43 INFO TaskSetManager: Finished task 14.0 in stage 7.0 (TID 43) in 472 ms on hadoop04 (executor 2) (16/100)
19/04/12 16:56:43 INFO TaskSetManager: Starting task 16.0 in stage 7.0 (TID 45, hadoop03, executor 1, partition 41, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:43 INFO TaskSetManager: Finished task 13.0 in stage 7.0 (TID 42) in 964 ms on hadoop03 (executor 1) (17/100)
19/04/12 16:56:45 INFO TaskSetManager: Starting task 17.0 in stage 7.0 (TID 46, hadoop03, executor 1, partition 42, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:46 INFO TaskSetManager: Starting task 18.0 in stage 7.0 (TID 47, hadoop04, executor 2, partition 43, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:46 INFO TaskSetManager: Finished task 15.0 in stage 7.0 (TID 44) in 2989 ms on hadoop04 (executor 2) (18/100)
19/04/12 16:56:46 INFO TaskSetManager: Finished task 16.0 in stage 7.0 (TID 45) in 3210 ms on hadoop03 (executor 1) (19/100)
19/04/12 16:56:46 INFO TaskSetManager: Starting task 19.0 in stage 7.0 (TID 48, hadoop03, executor 1, partition 44, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:46 INFO TaskSetManager: Finished task 17.0 in stage 7.0 (TID 46) in 1507 ms on hadoop03 (executor 1) (20/100)
19/04/12 16:56:47 INFO TaskSetManager: Starting task 20.0 in stage 7.0 (TID 49, hadoop04, executor 2, partition 45, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:47 INFO TaskSetManager: Finished task 18.0 in stage 7.0 (TID 47) in 1533 ms on hadoop04 (executor 2) (21/100)
19/04/12 16:56:48 INFO TaskSetManager: Starting task 21.0 in stage 7.0 (TID 50, hadoop04, executor 2, partition 46, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:48 INFO TaskSetManager: Finished task 20.0 in stage 7.0 (TID 49) in 715 ms on hadoop04 (executor 2) (22/100)
19/04/12 16:56:48 INFO TaskSetManager: Starting task 22.0 in stage 7.0 (TID 51, hadoop03, executor 1, partition 47, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:49 INFO TaskSetManager: Finished task 19.0 in stage 7.0 (TID 48) in 2132 ms on hadoop03 (executor 1) (23/100)
19/04/12 16:56:49 INFO TaskSetManager: Starting task 23.0 in stage 7.0 (TID 52, hadoop04, executor 2, partition 48, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:49 INFO TaskSetManager: Finished task 21.0 in stage 7.0 (TID 50) in 1332 ms on hadoop04 (executor 2) (24/100)
19/04/12 16:56:49 INFO TaskSetManager: Starting task 24.0 in stage 7.0 (TID 53, hadoop03, executor 1, partition 49, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:49 INFO TaskSetManager: Finished task 22.0 in stage 7.0 (TID 51) in 776 ms on hadoop03 (executor 1) (25/100)
19/04/12 16:56:50 INFO TaskSetManager: Starting task 25.0 in stage 7.0 (TID 54, hadoop04, executor 2, partition 50, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:50 INFO TaskSetManager: Finished task 23.0 in stage 7.0 (TID 52) in 1066 ms on hadoop04 (executor 2) (26/100)
19/04/12 16:56:50 INFO TaskSetManager: Starting task 26.0 in stage 7.0 (TID 55, hadoop03, executor 1, partition 51, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:50 INFO TaskSetManager: Finished task 24.0 in stage 7.0 (TID 53) in 698 ms on hadoop03 (executor 1) (27/100)
19/04/12 16:56:50 INFO TaskSetManager: Starting task 27.0 in stage 7.0 (TID 56, hadoop04, executor 2, partition 52, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:50 INFO TaskSetManager: Finished task 25.0 in stage 7.0 (TID 54) in 679 ms on hadoop04 (executor 2) (28/100)
19/04/12 16:56:50 INFO TaskSetManager: Starting task 28.0 in stage 7.0 (TID 57, hadoop03, executor 1, partition 53, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:51 INFO TaskSetManager: Finished task 26.0 in stage 7.0 (TID 55) in 1082 ms on hadoop03 (executor 1) (29/100)
19/04/12 16:56:51 INFO TaskSetManager: Starting task 29.0 in stage 7.0 (TID 58, hadoop04, executor 2, partition 54, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:51 INFO TaskSetManager: Finished task 27.0 in stage 7.0 (TID 56) in 731 ms on hadoop04 (executor 2) (30/100)
19/04/12 16:56:51 INFO TaskSetManager: Starting task 30.0 in stage 7.0 (TID 59, hadoop03, executor 1, partition 55, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:51 INFO TaskSetManager: Finished task 28.0 in stage 7.0 (TID 57) in 592 ms on hadoop03 (executor 1) (31/100)
19/04/12 16:56:51 INFO TaskSetManager: Starting task 31.0 in stage 7.0 (TID 60, hadoop04, executor 2, partition 56, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:51 INFO TaskSetManager: Finished task 29.0 in stage 7.0 (TID 58) in 557 ms on hadoop04 (executor 2) (32/100)
19/04/12 16:56:51 INFO TaskSetManager: Starting task 32.0 in stage 7.0 (TID 61, hadoop03, executor 1, partition 57, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:51 INFO TaskSetManager: Finished task 30.0 in stage 7.0 (TID 59) in 468 ms on hadoop03 (executor 1) (33/100)
19/04/12 16:56:51 INFO TaskSetManager: Starting task 33.0 in stage 7.0 (TID 62, hadoop04, executor 2, partition 58, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:52 INFO TaskSetManager: Finished task 31.0 in stage 7.0 (TID 60) in 241 ms on hadoop04 (executor 2) (34/100)
19/04/12 16:56:52 INFO TaskSetManager: Starting task 34.0 in stage 7.0 (TID 63, hadoop04, executor 2, partition 59, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:52 INFO TaskSetManager: Finished task 33.0 in stage 7.0 (TID 62) in 728 ms on hadoop04 (executor 2) (35/100)
19/04/12 16:56:52 INFO TaskSetManager: Starting task 35.0 in stage 7.0 (TID 64, hadoop03, executor 1, partition 60, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:52 INFO TaskSetManager: Finished task 32.0 in stage 7.0 (TID 61) in 804 ms on hadoop03 (executor 1) (36/100)
19/04/12 16:56:52 INFO TaskSetManager: Starting task 36.0 in stage 7.0 (TID 65, hadoop04, executor 2, partition 61, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:52 INFO TaskSetManager: Finished task 34.0 in stage 7.0 (TID 63) in 392 ms on hadoop04 (executor 2) (37/100)
19/04/12 16:56:53 INFO TaskSetManager: Starting task 37.0 in stage 7.0 (TID 66, hadoop03, executor 1, partition 62, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:53 INFO TaskSetManager: Finished task 35.0 in stage 7.0 (TID 64) in 320 ms on hadoop03 (executor 1) (38/100)
19/04/12 16:56:53 INFO TaskSetManager: Starting task 38.0 in stage 7.0 (TID 67, hadoop04, executor 2, partition 63, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:53 INFO TaskSetManager: Finished task 36.0 in stage 7.0 (TID 65) in 154 ms on hadoop04 (executor 2) (39/100)
19/04/12 16:56:53 INFO TaskSetManager: Starting task 39.0 in stage 7.0 (TID 68, hadoop03, executor 1, partition 64, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:53 INFO TaskSetManager: Finished task 37.0 in stage 7.0 (TID 66) in 297 ms on hadoop03 (executor 1) (40/100)
19/04/12 16:56:53 INFO TaskSetManager: Starting task 40.0 in stage 7.0 (TID 69, hadoop03, executor 1, partition 65, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:53 INFO TaskSetManager: Finished task 39.0 in stage 7.0 (TID 68) in 463 ms on hadoop03 (executor 1) (41/100)
19/04/12 16:56:53 INFO TaskSetManager: Starting task 41.0 in stage 7.0 (TID 70, hadoop04, executor 2, partition 66, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:53 INFO TaskSetManager: Finished task 38.0 in stage 7.0 (TID 67) in 672 ms on hadoop04 (executor 2) (42/100)
19/04/12 16:56:53 INFO TaskSetManager: Starting task 42.0 in stage 7.0 (TID 71, hadoop03, executor 1, partition 67, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:53 INFO TaskSetManager: Finished task 40.0 in stage 7.0 (TID 69) in 165 ms on hadoop03 (executor 1) (43/100)
19/04/12 16:56:55 INFO TaskSetManager: Starting task 43.0 in stage 7.0 (TID 72, hadoop03, executor 1, partition 68, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:55 INFO TaskSetManager: Finished task 42.0 in stage 7.0 (TID 71) in 1656 ms on hadoop03 (executor 1) (44/100)
19/04/12 16:56:55 INFO TaskSetManager: Starting task 44.0 in stage 7.0 (TID 73, hadoop03, executor 1, partition 69, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:55 INFO TaskSetManager: Finished task 43.0 in stage 7.0 (TID 72) in 534 ms on hadoop03 (executor 1) (45/100)
19/04/12 16:56:56 INFO TaskSetManager: Starting task 45.0 in stage 7.0 (TID 74, hadoop03, executor 1, partition 70, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:56 INFO TaskSetManager: Finished task 44.0 in stage 7.0 (TID 73) in 376 ms on hadoop03 (executor 1) (46/100)
19/04/12 16:56:56 INFO TaskSetManager: Starting task 46.0 in stage 7.0 (TID 75, hadoop04, executor 2, partition 71, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:56 INFO TaskSetManager: Finished task 41.0 in stage 7.0 (TID 70) in 2844 ms on hadoop04 (executor 2) (47/100)
19/04/12 16:56:56 INFO TaskSetManager: Starting task 47.0 in stage 7.0 (TID 76, hadoop03, executor 1, partition 72, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:56 INFO TaskSetManager: Finished task 45.0 in stage 7.0 (TID 74) in 444 ms on hadoop03 (executor 1) (48/100)
19/04/12 16:56:56 INFO TaskSetManager: Starting task 48.0 in stage 7.0 (TID 77, hadoop03, executor 1, partition 73, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:56 INFO TaskSetManager: Finished task 47.0 in stage 7.0 (TID 76) in 341 ms on hadoop03 (executor 1) (49/100)
19/04/12 16:56:57 INFO TaskSetManager: Starting task 49.0 in stage 7.0 (TID 78, hadoop03, executor 1, partition 74, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:57 INFO TaskSetManager: Finished task 48.0 in stage 7.0 (TID 77) in 169 ms on hadoop03 (executor 1) (50/100)
19/04/12 16:56:57 INFO TaskSetManager: Starting task 50.0 in stage 7.0 (TID 79, hadoop03, executor 1, partition 75, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:57 INFO TaskSetManager: Finished task 49.0 in stage 7.0 (TID 78) in 193 ms on hadoop03 (executor 1) (51/100)
19/04/12 16:56:57 INFO TaskSetManager: Starting task 51.0 in stage 7.0 (TID 80, hadoop04, executor 2, partition 76, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:57 INFO TaskSetManager: Finished task 46.0 in stage 7.0 (TID 75) in 1174 ms on hadoop04 (executor 2) (52/100)
19/04/12 16:56:57 INFO TaskSetManager: Starting task 52.0 in stage 7.0 (TID 81, hadoop03, executor 1, partition 77, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:57 INFO TaskSetManager: Starting task 53.0 in stage 7.0 (TID 82, hadoop04, executor 2, partition 78, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:57 INFO TaskSetManager: Finished task 50.0 in stage 7.0 (TID 79) in 337 ms on hadoop03 (executor 1) (53/100)
19/04/12 16:56:57 INFO TaskSetManager: Finished task 51.0 in stage 7.0 (TID 80) in 310 ms on hadoop04 (executor 2) (54/100)
19/04/12 16:56:58 INFO TaskSetManager: Starting task 54.0 in stage 7.0 (TID 83, hadoop03, executor 1, partition 79, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:58 INFO TaskSetManager: Finished task 52.0 in stage 7.0 (TID 81) in 761 ms on hadoop03 (executor 1) (55/100)
19/04/12 16:56:58 INFO TaskSetManager: Starting task 55.0 in stage 7.0 (TID 84, hadoop04, executor 2, partition 80, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:58 INFO TaskSetManager: Finished task 53.0 in stage 7.0 (TID 82) in 696 ms on hadoop04 (executor 2) (56/100)
19/04/12 16:56:58 INFO TaskSetManager: Starting task 56.0 in stage 7.0 (TID 85, hadoop03, executor 1, partition 81, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:58 INFO TaskSetManager: Finished task 54.0 in stage 7.0 (TID 83) in 691 ms on hadoop03 (executor 1) (57/100)
19/04/12 16:56:58 INFO TaskSetManager: Starting task 57.0 in stage 7.0 (TID 86, hadoop03, executor 1, partition 82, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:58 INFO TaskSetManager: Finished task 56.0 in stage 7.0 (TID 85) in 118 ms on hadoop03 (executor 1) (58/100)
19/04/12 16:56:58 INFO TaskSetManager: Starting task 58.0 in stage 7.0 (TID 87, hadoop04, executor 2, partition 83, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:58 INFO TaskSetManager: Finished task 55.0 in stage 7.0 (TID 84) in 447 ms on hadoop04 (executor 2) (59/100)
19/04/12 16:56:58 INFO TaskSetManager: Starting task 59.0 in stage 7.0 (TID 88, hadoop03, executor 1, partition 84, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:58 INFO TaskSetManager: Finished task 57.0 in stage 7.0 (TID 86) in 240 ms on hadoop03 (executor 1) (60/100)
19/04/12 16:56:59 INFO TaskSetManager: Starting task 60.0 in stage 7.0 (TID 89, hadoop04, executor 2, partition 85, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:59 INFO TaskSetManager: Finished task 58.0 in stage 7.0 (TID 87) in 325 ms on hadoop04 (executor 2) (61/100)
19/04/12 16:56:59 INFO TaskSetManager: Starting task 61.0 in stage 7.0 (TID 90, hadoop03, executor 1, partition 86, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:59 INFO TaskSetManager: Finished task 59.0 in stage 7.0 (TID 88) in 255 ms on hadoop03 (executor 1) (62/100)
19/04/12 16:56:59 INFO TaskSetManager: Starting task 62.0 in stage 7.0 (TID 91, hadoop04, executor 2, partition 87, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:59 INFO TaskSetManager: Finished task 60.0 in stage 7.0 (TID 89) in 147 ms on hadoop04 (executor 2) (63/100)
19/04/12 16:56:59 INFO TaskSetManager: Starting task 63.0 in stage 7.0 (TID 92, hadoop03, executor 1, partition 88, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:59 INFO TaskSetManager: Finished task 61.0 in stage 7.0 (TID 90) in 130 ms on hadoop03 (executor 1) (64/100)
19/04/12 16:56:59 INFO TaskSetManager: Starting task 64.0 in stage 7.0 (TID 93, hadoop03, executor 1, partition 89, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:59 INFO TaskSetManager: Finished task 63.0 in stage 7.0 (TID 92) in 110 ms on hadoop03 (executor 1) (65/100)
19/04/12 16:56:59 INFO TaskSetManager: Starting task 65.0 in stage 7.0 (TID 94, hadoop04, executor 2, partition 90, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:59 INFO TaskSetManager: Finished task 62.0 in stage 7.0 (TID 91) in 264 ms on hadoop04 (executor 2) (66/100)
19/04/12 16:56:59 INFO TaskSetManager: Starting task 66.0 in stage 7.0 (TID 95, hadoop03, executor 1, partition 91, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:59 INFO TaskSetManager: Finished task 64.0 in stage 7.0 (TID 93) in 165 ms on hadoop03 (executor 1) (67/100)
19/04/12 16:56:59 INFO TaskSetManager: Starting task 67.0 in stage 7.0 (TID 96, hadoop04, executor 2, partition 92, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:59 INFO TaskSetManager: Finished task 65.0 in stage 7.0 (TID 94) in 267 ms on hadoop04 (executor 2) (68/100)
19/04/12 16:56:59 INFO TaskSetManager: Starting task 68.0 in stage 7.0 (TID 97, hadoop03, executor 1, partition 93, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:59 INFO TaskSetManager: Finished task 66.0 in stage 7.0 (TID 95) in 222 ms on hadoop03 (executor 1) (69/100)
19/04/12 16:56:59 INFO TaskSetManager: Starting task 69.0 in stage 7.0 (TID 98, hadoop04, executor 2, partition 94, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:59 INFO TaskSetManager: Finished task 67.0 in stage 7.0 (TID 96) in 191 ms on hadoop04 (executor 2) (70/100)
19/04/12 16:56:59 INFO TaskSetManager: Starting task 70.0 in stage 7.0 (TID 99, hadoop03, executor 1, partition 95, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:59 INFO TaskSetManager: Finished task 68.0 in stage 7.0 (TID 97) in 173 ms on hadoop03 (executor 1) (71/100)
19/04/12 16:56:59 INFO TaskSetManager: Starting task 71.0 in stage 7.0 (TID 100, hadoop04, executor 2, partition 96, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:59 INFO TaskSetManager: Finished task 69.0 in stage 7.0 (TID 98) in 165 ms on hadoop04 (executor 2) (72/100)
19/04/12 16:56:59 INFO TaskSetManager: Starting task 72.0 in stage 7.0 (TID 101, hadoop03, executor 1, partition 97, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:59 INFO TaskSetManager: Finished task 70.0 in stage 7.0 (TID 99) in 181 ms on hadoop03 (executor 1) (73/100)
19/04/12 16:56:59 INFO TaskSetManager: Starting task 73.0 in stage 7.0 (TID 102, hadoop04, executor 2, partition 98, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:56:59 INFO TaskSetManager: Finished task 71.0 in stage 7.0 (TID 100) in 176 ms on hadoop04 (executor 2) (74/100)
19/04/12 16:57:00 INFO TaskSetManager: Starting task 74.0 in stage 7.0 (TID 103, hadoop04, executor 2, partition 99, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:00 INFO TaskSetManager: Finished task 73.0 in stage 7.0 (TID 102) in 103 ms on hadoop04 (executor 2) (75/100)
19/04/12 16:57:00 INFO TaskSetManager: Starting task 75.0 in stage 7.0 (TID 104, hadoop03, executor 1, partition 100, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:00 INFO TaskSetManager: Finished task 72.0 in stage 7.0 (TID 101) in 202 ms on hadoop03 (executor 1) (76/100)
19/04/12 16:57:00 INFO TaskSetManager: Starting task 76.0 in stage 7.0 (TID 105, hadoop04, executor 2, partition 101, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:00 INFO TaskSetManager: Finished task 74.0 in stage 7.0 (TID 103) in 159 ms on hadoop04 (executor 2) (77/100)
19/04/12 16:57:00 INFO TaskSetManager: Starting task 78.0 in stage 7.0 (TID 106, hadoop04, executor 2, partition 103, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:00 INFO TaskSetManager: Finished task 76.0 in stage 7.0 (TID 105) in 378 ms on hadoop04 (executor 2) (78/100)
19/04/12 16:57:00 INFO TaskSetManager: Starting task 79.0 in stage 7.0 (TID 107, hadoop03, executor 1, partition 104, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:00 INFO TaskSetManager: Finished task 75.0 in stage 7.0 (TID 104) in 551 ms on hadoop03 (executor 1) (79/100)
19/04/12 16:57:00 INFO TaskSetManager: Starting task 80.0 in stage 7.0 (TID 108, hadoop04, executor 2, partition 105, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:00 INFO TaskSetManager: Finished task 78.0 in stage 7.0 (TID 106) in 437 ms on hadoop04 (executor 2) (80/100)
19/04/12 16:57:01 INFO TaskSetManager: Starting task 81.0 in stage 7.0 (TID 109, hadoop03, executor 1, partition 106, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:01 INFO TaskSetManager: Finished task 79.0 in stage 7.0 (TID 107) in 621 ms on hadoop03 (executor 1) (81/100)
19/04/12 16:57:01 INFO TaskSetManager: Starting task 82.0 in stage 7.0 (TID 110, hadoop04, executor 2, partition 107, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:01 INFO TaskSetManager: Finished task 80.0 in stage 7.0 (TID 108) in 1254 ms on hadoop04 (executor 2) (82/100)
19/04/12 16:57:02 INFO TaskSetManager: Starting task 83.0 in stage 7.0 (TID 111, hadoop03, executor 1, partition 108, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:02 INFO TaskSetManager: Finished task 81.0 in stage 7.0 (TID 109) in 1316 ms on hadoop03 (executor 1) (83/100)
19/04/12 16:57:02 INFO TaskSetManager: Starting task 84.0 in stage 7.0 (TID 112, hadoop04, executor 2, partition 109, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:02 INFO TaskSetManager: Finished task 82.0 in stage 7.0 (TID 110) in 793 ms on hadoop04 (executor 2) (84/100)
19/04/12 16:57:02 INFO TaskSetManager: Starting task 86.0 in stage 7.0 (TID 113, hadoop03, executor 1, partition 111, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:02 INFO TaskSetManager: Finished task 83.0 in stage 7.0 (TID 111) in 780 ms on hadoop03 (executor 1) (85/100)
19/04/12 16:57:03 INFO TaskSetManager: Starting task 87.0 in stage 7.0 (TID 114, hadoop04, executor 2, partition 112, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:03 INFO TaskSetManager: Finished task 84.0 in stage 7.0 (TID 112) in 567 ms on hadoop04 (executor 2) (86/100)
19/04/12 16:57:03 INFO TaskSetManager: Starting task 88.0 in stage 7.0 (TID 115, hadoop03, executor 1, partition 113, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:03 INFO TaskSetManager: Finished task 86.0 in stage 7.0 (TID 113) in 689 ms on hadoop03 (executor 1) (87/100)
19/04/12 16:57:03 INFO TaskSetManager: Starting task 89.0 in stage 7.0 (TID 116, hadoop04, executor 2, partition 114, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:03 INFO TaskSetManager: Finished task 87.0 in stage 7.0 (TID 114) in 513 ms on hadoop04 (executor 2) (88/100)
19/04/12 16:57:03 INFO TaskSetManager: Starting task 90.0 in stage 7.0 (TID 117, hadoop03, executor 1, partition 115, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:03 INFO TaskSetManager: Finished task 88.0 in stage 7.0 (TID 115) in 432 ms on hadoop03 (executor 1) (89/100)
19/04/12 16:57:03 INFO TaskSetManager: Starting task 91.0 in stage 7.0 (TID 118, hadoop04, executor 2, partition 116, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:03 INFO TaskSetManager: Finished task 89.0 in stage 7.0 (TID 116) in 271 ms on hadoop04 (executor 2) (90/100)
19/04/12 16:57:03 INFO TaskSetManager: Starting task 92.0 in stage 7.0 (TID 119, hadoop03, executor 1, partition 117, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:03 INFO TaskSetManager: Starting task 93.0 in stage 7.0 (TID 120, hadoop04, executor 2, partition 118, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:03 INFO TaskSetManager: Finished task 91.0 in stage 7.0 (TID 118) in 129 ms on hadoop04 (executor 2) (91/100)
19/04/12 16:57:03 INFO TaskSetManager: Finished task 90.0 in stage 7.0 (TID 117) in 173 ms on hadoop03 (executor 1) (92/100)
19/04/12 16:57:03 INFO TaskSetManager: Starting task 94.0 in stage 7.0 (TID 121, hadoop04, executor 2, partition 119, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:03 INFO TaskSetManager: Finished task 93.0 in stage 7.0 (TID 120) in 139 ms on hadoop04 (executor 2) (93/100)
19/04/12 16:57:04 INFO TaskSetManager: Starting task 95.0 in stage 7.0 (TID 122, hadoop03, executor 1, partition 120, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:04 INFO TaskSetManager: Finished task 92.0 in stage 7.0 (TID 119) in 232 ms on hadoop03 (executor 1) (94/100)
19/04/12 16:57:04 INFO TaskSetManager: Starting task 96.0 in stage 7.0 (TID 123, hadoop04, executor 2, partition 121, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:04 INFO TaskSetManager: Finished task 94.0 in stage 7.0 (TID 121) in 110 ms on hadoop04 (executor 2) (95/100)
19/04/12 16:57:04 INFO TaskSetManager: Starting task 97.0 in stage 7.0 (TID 124, hadoop03, executor 1, partition 122, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:04 INFO TaskSetManager: Finished task 95.0 in stage 7.0 (TID 122) in 115 ms on hadoop03 (executor 1) (96/100)
19/04/12 16:57:04 INFO TaskSetManager: Starting task 98.0 in stage 7.0 (TID 125, hadoop04, executor 2, partition 123, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:04 INFO TaskSetManager: Finished task 96.0 in stage 7.0 (TID 123) in 136 ms on hadoop04 (executor 2) (97/100)
19/04/12 16:57:04 INFO TaskSetManager: Starting task 99.0 in stage 7.0 (TID 126, hadoop03, executor 1, partition 124, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:04 INFO TaskSetManager: Finished task 97.0 in stage 7.0 (TID 124) in 124 ms on hadoop03 (executor 1) (98/100)
19/04/12 16:57:04 INFO TaskSetManager: Finished task 98.0 in stage 7.0 (TID 125) in 140 ms on hadoop04 (executor 2) (99/100)
19/04/12 16:57:04 INFO TaskSetManager: Finished task 99.0 in stage 7.0 (TID 126) in 145 ms on hadoop03 (executor 1) (100/100)
19/04/12 16:57:04 INFO YarnScheduler: Removed TaskSet 7.0, whose tasks have all completed, from pool 
19/04/12 16:57:04 INFO DAGScheduler: ResultStage 7 (show at WindowFunctionTest.scala:45) finished in 26.819 s
19/04/12 16:57:04 INFO DAGScheduler: Job 3 finished: show at WindowFunctionTest.scala:45, took 27.209729 s
19/04/12 16:57:05 INFO SparkContext: Starting job: show at WindowFunctionTest.scala:45
19/04/12 16:57:05 INFO DAGScheduler: Got job 4 (show at WindowFunctionTest.scala:45) with 75 output partitions
19/04/12 16:57:05 INFO DAGScheduler: Final stage: ResultStage 9 (show at WindowFunctionTest.scala:45)
19/04/12 16:57:05 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 8)
19/04/12 16:57:05 INFO DAGScheduler: Missing parents: List()
19/04/12 16:57:05 INFO DAGScheduler: Submitting ResultStage 9 (MapPartitionsRDD[8] at show at WindowFunctionTest.scala:45), which has no missing parents
19/04/12 16:57:05 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 18.4 KB, free 447.2 MB)
19/04/12 16:57:05 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 8.8 KB, free 447.2 MB)
19/04/12 16:57:05 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on hadoop04:42715 (size: 8.8 KB, free: 447.3 MB)
19/04/12 16:57:05 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1161
19/04/12 16:57:05 INFO DAGScheduler: Submitting 75 missing tasks from ResultStage 9 (MapPartitionsRDD[8] at show at WindowFunctionTest.scala:45) (first 15 tasks are for partitions Vector(125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139))
19/04/12 16:57:05 INFO YarnScheduler: Adding task set 9.0 with 75 tasks
19/04/12 16:57:05 INFO TaskSetManager: Starting task 0.0 in stage 9.0 (TID 127, hadoop04, executor 2, partition 125, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:05 INFO TaskSetManager: Starting task 1.0 in stage 9.0 (TID 128, hadoop03, executor 1, partition 126, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:05 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on hadoop04:38267 (size: 8.8 KB, free: 366.3 MB)
19/04/12 16:57:06 INFO TaskSetManager: Starting task 2.0 in stage 9.0 (TID 129, hadoop04, executor 2, partition 127, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:06 INFO TaskSetManager: Finished task 0.0 in stage 9.0 (TID 127) in 398 ms on hadoop04 (executor 2) (1/75)
19/04/12 16:57:06 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on hadoop03:45104 (size: 8.8 KB, free: 366.3 MB)
19/04/12 16:57:06 INFO TaskSetManager: Starting task 3.0 in stage 9.0 (TID 130, hadoop04, executor 2, partition 128, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:06 INFO TaskSetManager: Finished task 2.0 in stage 9.0 (TID 129) in 259 ms on hadoop04 (executor 2) (2/75)
19/04/12 16:57:06 INFO TaskSetManager: Starting task 4.0 in stage 9.0 (TID 131, hadoop03, executor 1, partition 129, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:06 INFO TaskSetManager: Finished task 1.0 in stage 9.0 (TID 128) in 796 ms on hadoop03 (executor 1) (3/75)
19/04/12 16:57:06 INFO TaskSetManager: Starting task 5.0 in stage 9.0 (TID 132, hadoop04, executor 2, partition 130, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:06 INFO TaskSetManager: Finished task 3.0 in stage 9.0 (TID 130) in 277 ms on hadoop04 (executor 2) (4/75)
19/04/12 16:57:06 INFO TaskSetManager: Starting task 6.0 in stage 9.0 (TID 133, hadoop03, executor 1, partition 131, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:06 INFO TaskSetManager: Finished task 4.0 in stage 9.0 (TID 131) in 188 ms on hadoop03 (executor 1) (5/75)
19/04/12 16:57:06 INFO TaskSetManager: Starting task 7.0 in stage 9.0 (TID 134, hadoop03, executor 1, partition 132, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:06 INFO TaskSetManager: Finished task 6.0 in stage 9.0 (TID 133) in 173 ms on hadoop03 (executor 1) (6/75)
19/04/12 16:57:06 INFO TaskSetManager: Starting task 8.0 in stage 9.0 (TID 135, hadoop04, executor 2, partition 133, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:06 INFO TaskSetManager: Finished task 5.0 in stage 9.0 (TID 132) in 289 ms on hadoop04 (executor 2) (7/75)
19/04/12 16:57:07 INFO TaskSetManager: Starting task 9.0 in stage 9.0 (TID 136, hadoop03, executor 1, partition 134, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:07 INFO TaskSetManager: Finished task 7.0 in stage 9.0 (TID 134) in 206 ms on hadoop03 (executor 1) (8/75)
19/04/12 16:57:07 INFO TaskSetManager: Starting task 10.0 in stage 9.0 (TID 137, hadoop04, executor 2, partition 135, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:07 INFO TaskSetManager: Finished task 8.0 in stage 9.0 (TID 135) in 187 ms on hadoop04 (executor 2) (9/75)
19/04/12 16:57:07 INFO TaskSetManager: Starting task 11.0 in stage 9.0 (TID 138, hadoop03, executor 1, partition 136, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:07 INFO TaskSetManager: Starting task 12.0 in stage 9.0 (TID 139, hadoop04, executor 2, partition 137, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:07 INFO TaskSetManager: Finished task 9.0 in stage 9.0 (TID 136) in 218 ms on hadoop03 (executor 1) (10/75)
19/04/12 16:57:07 INFO TaskSetManager: Starting task 13.0 in stage 9.0 (TID 140, hadoop03, executor 1, partition 138, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:07 INFO TaskSetManager: Finished task 11.0 in stage 9.0 (TID 138) in 102 ms on hadoop03 (executor 1) (11/75)
19/04/12 16:57:07 INFO TaskSetManager: Finished task 10.0 in stage 9.0 (TID 137) in 232 ms on hadoop04 (executor 2) (12/75)
19/04/12 16:57:07 INFO TaskSetManager: Starting task 14.0 in stage 9.0 (TID 141, hadoop03, executor 1, partition 139, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:07 INFO TaskSetManager: Finished task 13.0 in stage 9.0 (TID 140) in 96 ms on hadoop03 (executor 1) (13/75)
19/04/12 16:57:07 INFO TaskSetManager: Starting task 15.0 in stage 9.0 (TID 142, hadoop04, executor 2, partition 140, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:07 INFO TaskSetManager: Finished task 12.0 in stage 9.0 (TID 139) in 200 ms on hadoop04 (executor 2) (14/75)
19/04/12 16:57:07 INFO TaskSetManager: Starting task 16.0 in stage 9.0 (TID 143, hadoop03, executor 1, partition 141, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:07 INFO TaskSetManager: Finished task 14.0 in stage 9.0 (TID 141) in 156 ms on hadoop03 (executor 1) (15/75)
19/04/12 16:57:07 INFO TaskSetManager: Starting task 17.0 in stage 9.0 (TID 144, hadoop03, executor 1, partition 142, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:07 INFO TaskSetManager: Finished task 16.0 in stage 9.0 (TID 143) in 234 ms on hadoop03 (executor 1) (16/75)
19/04/12 16:57:07 INFO TaskSetManager: Starting task 18.0 in stage 9.0 (TID 145, hadoop04, executor 2, partition 143, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:07 INFO TaskSetManager: Finished task 15.0 in stage 9.0 (TID 142) in 453 ms on hadoop04 (executor 2) (17/75)
19/04/12 16:57:07 INFO TaskSetManager: Starting task 19.0 in stage 9.0 (TID 146, hadoop03, executor 1, partition 144, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:07 INFO TaskSetManager: Finished task 17.0 in stage 9.0 (TID 144) in 273 ms on hadoop03 (executor 1) (18/75)
19/04/12 16:57:08 INFO TaskSetManager: Starting task 20.0 in stage 9.0 (TID 147, hadoop04, executor 2, partition 145, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:08 INFO TaskSetManager: Finished task 18.0 in stage 9.0 (TID 145) in 220 ms on hadoop04 (executor 2) (19/75)
19/04/12 16:57:08 INFO TaskSetManager: Starting task 21.0 in stage 9.0 (TID 148, hadoop03, executor 1, partition 146, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:08 INFO TaskSetManager: Finished task 19.0 in stage 9.0 (TID 146) in 178 ms on hadoop03 (executor 1) (20/75)
19/04/12 16:57:08 INFO TaskSetManager: Starting task 22.0 in stage 9.0 (TID 149, hadoop04, executor 2, partition 147, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:08 INFO TaskSetManager: Finished task 20.0 in stage 9.0 (TID 147) in 313 ms on hadoop04 (executor 2) (21/75)
19/04/12 16:57:08 INFO TaskSetManager: Starting task 23.0 in stage 9.0 (TID 150, hadoop03, executor 1, partition 148, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:08 INFO TaskSetManager: Finished task 21.0 in stage 9.0 (TID 148) in 279 ms on hadoop03 (executor 1) (22/75)
19/04/12 16:57:08 INFO TaskSetManager: Starting task 24.0 in stage 9.0 (TID 151, hadoop04, executor 2, partition 149, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:08 INFO TaskSetManager: Finished task 22.0 in stage 9.0 (TID 149) in 371 ms on hadoop04 (executor 2) (23/75)
19/04/12 16:57:08 INFO TaskSetManager: Starting task 25.0 in stage 9.0 (TID 152, hadoop03, executor 1, partition 150, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:08 INFO TaskSetManager: Finished task 23.0 in stage 9.0 (TID 150) in 291 ms on hadoop03 (executor 1) (24/75)
19/04/12 16:57:08 INFO TaskSetManager: Starting task 26.0 in stage 9.0 (TID 153, hadoop04, executor 2, partition 151, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:08 INFO TaskSetManager: Finished task 24.0 in stage 9.0 (TID 151) in 321 ms on hadoop04 (executor 2) (25/75)
19/04/12 16:57:08 INFO TaskSetManager: Starting task 27.0 in stage 9.0 (TID 154, hadoop03, executor 1, partition 152, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:08 INFO TaskSetManager: Finished task 25.0 in stage 9.0 (TID 152) in 385 ms on hadoop03 (executor 1) (26/75)
19/04/12 16:57:08 INFO TaskSetManager: Starting task 28.0 in stage 9.0 (TID 155, hadoop03, executor 1, partition 153, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 29.0 in stage 9.0 (TID 156, hadoop04, executor 2, partition 154, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 26.0 in stage 9.0 (TID 153) in 386 ms on hadoop04 (executor 2) (27/75)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 27.0 in stage 9.0 (TID 154) in 332 ms on hadoop03 (executor 1) (28/75)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 30.0 in stage 9.0 (TID 157, hadoop03, executor 1, partition 155, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 28.0 in stage 9.0 (TID 155) in 199 ms on hadoop03 (executor 1) (29/75)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 31.0 in stage 9.0 (TID 158, hadoop04, executor 2, partition 156, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 29.0 in stage 9.0 (TID 156) in 303 ms on hadoop04 (executor 2) (30/75)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 32.0 in stage 9.0 (TID 159, hadoop04, executor 2, partition 157, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 31.0 in stage 9.0 (TID 158) in 94 ms on hadoop04 (executor 2) (31/75)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 33.0 in stage 9.0 (TID 160, hadoop03, executor 1, partition 158, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 30.0 in stage 9.0 (TID 157) in 276 ms on hadoop03 (executor 1) (32/75)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 34.0 in stage 9.0 (TID 161, hadoop04, executor 2, partition 159, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 32.0 in stage 9.0 (TID 159) in 171 ms on hadoop04 (executor 2) (33/75)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 35.0 in stage 9.0 (TID 162, hadoop03, executor 1, partition 160, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 33.0 in stage 9.0 (TID 160) in 182 ms on hadoop03 (executor 1) (34/75)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 36.0 in stage 9.0 (TID 163, hadoop03, executor 1, partition 161, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 37.0 in stage 9.0 (TID 164, hadoop04, executor 2, partition 162, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 38.0 in stage 9.0 (TID 165, hadoop04, executor 2, partition 163, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 37.0 in stage 9.0 (TID 164) in 52 ms on hadoop04 (executor 2) (35/75)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 34.0 in stage 9.0 (TID 161) in 191 ms on hadoop04 (executor 2) (36/75)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 35.0 in stage 9.0 (TID 162) in 206 ms on hadoop03 (executor 1) (37/75)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 39.0 in stage 9.0 (TID 166, hadoop03, executor 1, partition 164, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 40.0 in stage 9.0 (TID 167, hadoop04, executor 2, partition 165, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 38.0 in stage 9.0 (TID 165) in 108 ms on hadoop04 (executor 2) (38/75)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 41.0 in stage 9.0 (TID 168, hadoop04, executor 2, partition 166, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 36.0 in stage 9.0 (TID 163) in 218 ms on hadoop03 (executor 1) (39/75)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 40.0 in stage 9.0 (TID 167) in 88 ms on hadoop04 (executor 2) (40/75)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 42.0 in stage 9.0 (TID 169, hadoop03, executor 1, partition 167, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 39.0 in stage 9.0 (TID 166) in 112 ms on hadoop03 (executor 1) (41/75)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 43.0 in stage 9.0 (TID 170, hadoop03, executor 1, partition 168, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 42.0 in stage 9.0 (TID 169) in 75 ms on hadoop03 (executor 1) (42/75)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 44.0 in stage 9.0 (TID 171, hadoop04, executor 2, partition 169, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 41.0 in stage 9.0 (TID 168) in 162 ms on hadoop04 (executor 2) (43/75)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 45.0 in stage 9.0 (TID 172, hadoop03, executor 1, partition 170, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 46.0 in stage 9.0 (TID 173, hadoop04, executor 2, partition 171, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 43.0 in stage 9.0 (TID 170) in 96 ms on hadoop03 (executor 1) (44/75)
19/04/12 16:57:09 INFO TaskSetManager: Starting task 47.0 in stage 9.0 (TID 174, hadoop03, executor 1, partition 172, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 44.0 in stage 9.0 (TID 171) in 75 ms on hadoop04 (executor 2) (45/75)
19/04/12 16:57:09 INFO TaskSetManager: Finished task 45.0 in stage 9.0 (TID 172) in 49 ms on hadoop03 (executor 1) (46/75)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 48.0 in stage 9.0 (TID 175, hadoop03, executor 1, partition 173, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 47.0 in stage 9.0 (TID 174) in 85 ms on hadoop03 (executor 1) (47/75)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 49.0 in stage 9.0 (TID 176, hadoop04, executor 2, partition 174, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 50.0 in stage 9.0 (TID 177, hadoop03, executor 1, partition 175, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 51.0 in stage 9.0 (TID 178, hadoop03, executor 1, partition 176, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 50.0 in stage 9.0 (TID 177) in 38 ms on hadoop03 (executor 1) (48/75)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 46.0 in stage 9.0 (TID 173) in 141 ms on hadoop04 (executor 2) (49/75)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 52.0 in stage 9.0 (TID 179, hadoop03, executor 1, partition 177, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 48.0 in stage 9.0 (TID 175) in 123 ms on hadoop03 (executor 1) (50/75)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 51.0 in stage 9.0 (TID 178) in 57 ms on hadoop03 (executor 1) (51/75)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 53.0 in stage 9.0 (TID 180, hadoop04, executor 2, partition 178, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 49.0 in stage 9.0 (TID 176) in 114 ms on hadoop04 (executor 2) (52/75)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 54.0 in stage 9.0 (TID 181, hadoop03, executor 1, partition 179, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 55.0 in stage 9.0 (TID 182, hadoop03, executor 1, partition 180, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 56.0 in stage 9.0 (TID 183, hadoop04, executor 2, partition 181, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 52.0 in stage 9.0 (TID 179) in 159 ms on hadoop03 (executor 1) (53/75)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 57.0 in stage 9.0 (TID 184, hadoop03, executor 1, partition 182, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 54.0 in stage 9.0 (TID 181) in 110 ms on hadoop03 (executor 1) (54/75)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 58.0 in stage 9.0 (TID 185, hadoop04, executor 2, partition 183, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 53.0 in stage 9.0 (TID 180) in 165 ms on hadoop04 (executor 2) (55/75)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 55.0 in stage 9.0 (TID 182) in 262 ms on hadoop03 (executor 1) (56/75)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 59.0 in stage 9.0 (TID 186, hadoop03, executor 1, partition 184, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 60.0 in stage 9.0 (TID 187, hadoop04, executor 2, partition 185, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 56.0 in stage 9.0 (TID 183) in 331 ms on hadoop04 (executor 2) (57/75)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 57.0 in stage 9.0 (TID 184) in 301 ms on hadoop03 (executor 1) (58/75)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 58.0 in stage 9.0 (TID 185) in 289 ms on hadoop04 (executor 2) (59/75)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 61.0 in stage 9.0 (TID 188, hadoop03, executor 1, partition 186, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 59.0 in stage 9.0 (TID 186) in 151 ms on hadoop03 (executor 1) (60/75)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 62.0 in stage 9.0 (TID 189, hadoop04, executor 2, partition 187, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 60.0 in stage 9.0 (TID 187) in 197 ms on hadoop04 (executor 2) (61/75)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 63.0 in stage 9.0 (TID 190, hadoop03, executor 1, partition 188, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 61.0 in stage 9.0 (TID 188) in 114 ms on hadoop03 (executor 1) (62/75)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 64.0 in stage 9.0 (TID 191, hadoop04, executor 2, partition 189, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 62.0 in stage 9.0 (TID 189) in 148 ms on hadoop04 (executor 2) (63/75)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 65.0 in stage 9.0 (TID 192, hadoop03, executor 1, partition 190, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 63.0 in stage 9.0 (TID 190) in 113 ms on hadoop03 (executor 1) (64/75)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 66.0 in stage 9.0 (TID 193, hadoop03, executor 1, partition 191, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Starting task 67.0 in stage 9.0 (TID 194, hadoop04, executor 2, partition 192, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 64.0 in stage 9.0 (TID 191) in 129 ms on hadoop04 (executor 2) (65/75)
19/04/12 16:57:10 INFO TaskSetManager: Finished task 65.0 in stage 9.0 (TID 192) in 105 ms on hadoop03 (executor 1) (66/75)
19/04/12 16:57:11 INFO TaskSetManager: Starting task 68.0 in stage 9.0 (TID 195, hadoop03, executor 1, partition 193, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:11 INFO TaskSetManager: Finished task 66.0 in stage 9.0 (TID 193) in 126 ms on hadoop03 (executor 1) (67/75)
19/04/12 16:57:11 INFO TaskSetManager: Starting task 69.0 in stage 9.0 (TID 196, hadoop04, executor 2, partition 194, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:11 INFO TaskSetManager: Finished task 67.0 in stage 9.0 (TID 194) in 131 ms on hadoop04 (executor 2) (68/75)
19/04/12 16:57:11 INFO TaskSetManager: Starting task 70.0 in stage 9.0 (TID 197, hadoop04, executor 2, partition 195, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:11 INFO TaskSetManager: Finished task 69.0 in stage 9.0 (TID 196) in 217 ms on hadoop04 (executor 2) (69/75)
19/04/12 16:57:11 INFO TaskSetManager: Starting task 71.0 in stage 9.0 (TID 198, hadoop03, executor 1, partition 196, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:11 INFO TaskSetManager: Finished task 68.0 in stage 9.0 (TID 195) in 301 ms on hadoop03 (executor 1) (70/75)
19/04/12 16:57:11 INFO TaskSetManager: Starting task 72.0 in stage 9.0 (TID 199, hadoop04, executor 2, partition 197, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:11 INFO TaskSetManager: Finished task 70.0 in stage 9.0 (TID 197) in 270 ms on hadoop04 (executor 2) (71/75)
19/04/12 16:57:11 INFO TaskSetManager: Starting task 73.0 in stage 9.0 (TID 200, hadoop03, executor 1, partition 198, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:11 INFO TaskSetManager: Finished task 71.0 in stage 9.0 (TID 198) in 256 ms on hadoop03 (executor 1) (72/75)
19/04/12 16:57:11 INFO TaskSetManager: Starting task 74.0 in stage 9.0 (TID 201, hadoop04, executor 2, partition 199, PROCESS_LOCAL, 7778 bytes)
19/04/12 16:57:11 INFO TaskSetManager: Finished task 72.0 in stage 9.0 (TID 199) in 282 ms on hadoop04 (executor 2) (73/75)
19/04/12 16:57:11 INFO TaskSetManager: Finished task 73.0 in stage 9.0 (TID 200) in 214 ms on hadoop03 (executor 1) (74/75)
19/04/12 16:57:11 INFO TaskSetManager: Finished task 74.0 in stage 9.0 (TID 201) in 104 ms on hadoop04 (executor 2) (75/75)
19/04/12 16:57:11 INFO YarnScheduler: Removed TaskSet 9.0, whose tasks have all completed, from pool 
19/04/12 16:57:11 INFO DAGScheduler: ResultStage 9 (show at WindowFunctionTest.scala:45) finished in 6.626 s
19/04/12 16:57:11 INFO DAGScheduler: Job 4 finished: show at WindowFunctionTest.scala:45, took 6.700226 s
19/04/12 16:57:14 INFO SparkUI: Stopped Spark web UI at http://hadoop04:4040
19/04/12 16:57:14 INFO YarnClientSchedulerBackend: Interrupting monitor thread
19/04/12 16:57:15 INFO YarnClientSchedulerBackend: Shutting down all executors
19/04/12 16:57:15 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
19/04/12 16:57:15 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
19/04/12 16:57:15 INFO YarnClientSchedulerBackend: Stopped
19/04/12 16:57:16 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/04/12 16:57:17 INFO MemoryStore: MemoryStore cleared
19/04/12 16:57:17 INFO BlockManager: BlockManager stopped
19/04/12 16:57:17 INFO BlockManagerMaster: BlockManagerMaster stopped
19/04/12 16:57:18 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/04/12 16:57:18 INFO SparkContext: Successfully stopped SparkContext
19/04/12 16:57:19 INFO ShutdownHookManager: Shutdown hook called
19/04/12 16:57:19 INFO ShutdownHookManager: Deleting directory /tmp/spark-f3533ae6-b45b-4a1f-8119-0c12775c5a33
+-----+----------+--------+---------+
| site|      date|user_cnt|MovingAvg|
+-----+----------+--------+---------+
|站点1|2018-01-01|      50|     47.5|
|站点1|2018-01-02|      45|     50.0|
|站点1|2018-01-03|      55|     50.0|
|站点2|2018-01-01|      25|     27.0|
|站点2|2018-01-02|      29|     27.0|
|站点2|2018-01-03|      27|     28.0|
+-----+----------+--------+---------+

Disconnected from the target VM, address: '127.0.0.1:45315', transport: 'socket'

Process finished with exit code 0

至此,IDEA已经能连接上yarn集群。

开启远程yarn client debug调试

第一步

将我们的应用程序打包(在hadoop05机器)，发送到hadoop01机器，并在hadoop01上启动我们的应用程序,具体如下： 

1
2
3
4
5
mkdir -p /home/hadoop/worker/sparklearning/target/   

scp -r sparklearning-1.0-SNAPSHOT.jar  hadoop@hadoop01:/home/hadoop/worker/sparklearning/target/

scp -r sparklearning-1.0-SNAPSHOT.jar  hadoop@hadoop01:/home/hadoop/worker/sparklearning/target/

在hadoop01提交spark应用

1
2
3
4
5
6
7
8
9
./bin/spark-submit --class com.xh.spark.sql.practice.JDSaleQuotaRevert \
--master yarn \
--deploy-mode client \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 6 \
--driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005" \
--conf "spark.executor.extraJavaOptions=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005" \
/home/hadoop/worker/sparklearning/target/sparklearning-1.0-SNAPSHOT.jar

第二步

在idea中Edit Configurations ——> 点击+号 ——>Remote —— Configuration ——> Debugger Mode (用Attach to remote JVM)——>host(hadoop01)
——> Port(默认5005，也可以选择使用未使用的端口，比如5656) ——> Command line arguments for remote JVN (-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005)

然后点击小虫子图标就可以run程序了！！！！

问题解决分享

Could not parse Master URL

解决：
在pom.xml中添加如下依赖

1
2
3
4
5
<dependency>
         <groupId>org.apache.spark</groupId>
         <artifactId>spark-yarn_${scala.binary.version}</artifactId>
         <version>${spark.version}</version>
</dependency>

Name node is in safe mode

解决：
在hadoop所在linux环境输入如下命令

1
hdfs dfsadmin -safemode leave

Exception in thread “main” org.apache.hadoop.security.AccessControlException:

Exception in thread “main” org.apache.hadoop.security.AccessControlException: Permission denied: user=deeplearning, access=WRITE, inode=”/user/deeplearning/.sparkStaging/application_1554947367832_0002”:hadoop:supergroup:drwxr-xr-x

解决：
在hdfs-site.xml添加如下配置

1
2
3
4
 <property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>

Diagnostics: Container [pid=3293,containerID=container_e12_1555047553207_0002_02_000001] is running beyond virtual memory limits. Current usage: 116.1 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.

解决： 在yarn-site.xml文件中添加如下配置

1
2
3
4
5
6
7
8
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>
<property>
   <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>5</value>
</property>

问题解决参考文章

https://stackoverflow.com/questions/41054700/could-not-parse-master-url

https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

http://www.cnblogs.com/lisi2016/p/6863923.html

致谢!

如有遇到问题，将问题发送至本人邮箱(t_spider@aliyun.com)。欢迎大家一起讨论问题！

本文标题:idea中Spark on yarn 远程Debug Spark 源码(Yarn Client+attach 模式)

文章作者:tang

发布时间:2019年07月02日 - 06:07

最后更新:2019年07月02日 - 06:07

原始链接:https://tgluon.github.io/2019/07/02/idea中Spark on yarn 远程Debug Spark 源码(Yarn Client+attach 模式)/

许可协议: 署名-非商业性使用-禁止演绎 4.0 国际 转载请保留原文链接及作者。

-------------本文结束感谢您的阅读-------------
分享到： 收藏夹 复制网址 邮件 微信 QQ空间 腾讯微博 豆瓣 一键分享 更多